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Abstract 

The  US  Air  Force  has  determined  that  improving  the  way  we  manage  our  supply 
chain  is  key  to  reducing  operating  costs  and  subsequently  providing  better  support  to  the 
warfighter.  One  method  of  improving  supply  chain  management  is  to  streamline  the  way 
suppliers  are  evaluated.  This  study  compared  the  efforts  of  private  industry  with 
government  program  offices  in  terms  of  supplier  evaluation  techniques.  The  expert  panel 
from  private  companies  was  identified  because  of  their  association  with  the  aerospace 
industry.  Government  employees  were  identified  because  of  their  recent  contract  pre¬ 
award  and  post-award  experience.  Input  was  received  through  the  use  of  open-ended 
interview  questions  and  was  later  analyzed  for  content.  Pattern  matching  analysis  was 
used  to  determine  the  best  practices  of  private  and  government  entities  and  to  determine 
the  differences  between  government  and  private  industry  in  terms  of  their  supplier 
evaluation  techniques.  The  findings  of  the  study  did  not  mention  the  names  of  those 
interviewed  but  rather  the  organization  they  were  associated  with. 
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DETERMINING  THE  VALUE  OF  AUTOMATION  IN  COMMERCIAL  AND  USAF 

SUPPLIER  EVALUATION  SYSTEMS 


I.  Introduction 


Chapter  Overview 

This  chapter  describes  the  fundamental  motivation  for  my  research.  It  provides 
background  information  to  explain  the  history  of  the  subject,  an  explanation  of  the 
research  objectives,  the  primary  research  questions,  and  the  investigative  questions.  It 
concludes  with  a  brief  description  of  the  methodology  used  and  the  scope  and  limitations 
of  the  study. 

Background 

The  development  of  automated  supplier  rating  systems  has  gained  popularity  in 
the  civilian  marketplace  over  the  last  few  years.  Based  on  the  increase  of  electronic 
media  in  the  business-to-business  environment  it  has  become  essential  to  modify 
traditional  supply  chain  management  (SCM)  practices  in  the  commercial  sector.  The 
USAF  acquisition  community  is  currently  studying  Purchasing  and  Supply  Chain 
Management  (PSCM)  as  a  means  to  improve  its  own  supply  chain.  One  of  the  ultimate 
goals  of  PSCM  is  to  improve  the  supplier  selection  process.  Two  suppliers,  which  sell 
the  same  product  with  the  same  specifications  and  terms,  rarely  perfonn  at  the  same  level 
(Smith,  2000:40).  Many  commercial  activities  have  implemented  rating  scales  (e.g., 
gold,  silver  bronze)  to  evaluate  supplier  performance  in  a  more  timely  fashion. 
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Problem 


My  research  focuses  on  four  of  PSCM’s  13  key  tenets.  They  include:  1)  the 
automation  of  routine  activities,  2)  better  management  of  key  suppliers,  3)  understanding 
the  supply  chain,  and  4)  rationalizing  the  supply  base.  It  appears  that  the  private  sector 
has  devised  many  new  strategies  for  success.  The  use  of  automated  supplier  evaluation 
techniques  continues  to  increase  in  private  industry.  For  the  most  part,  the  USAF 
acquisition  community  continues  to  use  time-consuming  evaluation  techniques  in 
assessing  a  supplier’s  performance.  This  research  intends  to  examine  why  such  a  gap 
exists  and  provides  recommendations  on  how  to  close  it. 

Research  Objectives 

There  are  five  primary  objectives  in  my  research:  1)  comparing  supplier 
evaluation  methods  of  commercial  companies  with  that  of  the  United  States  Air  Force 
(USAF)  acquisition  community,  2)  investigating  the  success  of  automated  evaluations  in 
the  commercial  sector,  3)  detennining  best  practices  for  evaluating  suppliers  in  the 
commercial  market,  4)  determining  what  ratings  can  be  quantified  during  an  evaluation, 
and  5)  detennining  if  the  Federal  Acquisition  Regulation  (FAR)  will  prohibit  USAF 
acquisition  personnel  from  transforming  the  supplier  evaluation  process. 

Research/Investigative  Questions 

This  thesis  presents  an  alternative  to  the  current  process.  The  ultimate  goal  is  to 
explore  whether  “commercial”  supplier  evaluation  methods  can  replace  the  USAF 
acquisition  pre-award  and  post- award  processes.  Further,  the  objective  is  to  determine 
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which  commercial  evaluation  methods  are  most  successful  in  selecting  the  most 
competent  and  timely  suppliers. 

My  thesis  research  question  is: 

Can  USAF  acquisition  personnel  use  supplier  ratings  in  lieu  of  conducting  award-fee 
boards  or  past  performance  evaluations? 

The  investigative  questions  I  will  attempt  to  answer  are: 

1.  What  are  the  similarities  between  USAF  rating  categories  and  commercial  rating 
categories?  (These  are  the  factors  which  are  used  in  award-fee  detenninations  and  past 
performance  evaluations) 

2.  Can  cost  avoidance  savings  be  quantified  if  either  were  replaced? 

3.  What  acquisition  reform  initiatives  or  regulations  would  aid  or  hinder  USAF 
application? 

Methodology 

This  research  is  being  conducted  as  a  case  study.  The  case  study  includes  content 
analysis  of  the  interview  responses  of  eight  senior  managers  from  commercial  companies 
in  the  aerospace  or  air  transportation  industries,  and  two  senior  managers  from 
government  program  offices.  The  interview  instrument  was  devised  of  questions  which 
map  to  the  research  and  investigative  questions  mentioned  previously.  Of  the  3 1 
questions,  2 1  were  asked  to  both  government  personnel  and  company  representatives,  6 
were  asked  to  commercial  companies  only,  and  4  were  asked  to  government  only. 

The  content  analysis  was  aided  by  the  statistical  software  StatPac.  After  interviewing  all 
ten  personnel,  the  responses  were  transcribed  and  transferred  into  StatPac.  The  statistical 
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information  from  StatPac  provided  the  framework  for  the  conclusions  and 
recommendations  presented  in  chapter  5. 

Scope  and  Limitations  of  the  Research 

A  case  study  limits  the  finding  to  the  cases  under  study.  In  this  study  only 
program  offices  from  Aeronautical  System  Center,  Wright-Patterson  Air  Force  Base,  OH 
were  interviewed.  Many  of  the  generalizations  about  USAF  acquisition  become  difficult 
when  trying  to  distinguish  between  product  center,  depot,  and  operational  buying  units. 
The  findings  of  my  research  hope  to  encourage  innovative  thinking  and  more  efficient 
methods  for  evaluating  suppliers. 
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II.  Literature  Review 


Effort  Without  Stratification 

Pre-award  and  post-award  supplier  evaluations  require  extensive  amounts  of  time 
for  USAF  acquisition  personnel.  During  the  pre-award  evaluation  period,  contracting 
officers  and  members  of  the  past  perfonnance  evaluation  team  send  out  surveys  and 
research  contractor  past  perfonnance  data  to  determine  which  offerors  are  most  capable 
of  meeting  the  government  requirement.  This  process  can  take  as  long  as  six  months. 
The  current  processes  for  pre-award  and  post-award  collection  are  too  cumbersome  in 
today’s  acquisition  environment  (Beasoleil,  2000:28).  More  often  than  not,  the 
performance  risk  assessment  team  (PRAG),  which  is  responsible  for  pre-award 
evaluations  during  source  selections,  does  not  establish  discriminators  between  offerors. 
As  a  result,  the  PRAG  team  is  overshadowed  by  the  technical  and  cost  portions  of  the 
source  selection.  Businesses  that  deal  with  the  federal  government  feel  that  the  Federal 
Acquisition  Regulation  allows  too  much  latitude  to  evaluators,  resulting  in  inconsistent 
evaluation  factors  among  federal  agencies  (Clipsham,  1998:31).  Additionally,  during 
post-award  reviews,  the  Award-fee  Board  (AFRB),  which  often  includes  the  contracting 
officer,  the  program  manager,  and  the  logistics  manager,  requires  continuous  monitoring 
of  the  contractor/supplier  in  order  to  determine  the  award-fee  earned.  This  process 
typically  takes  30  to  60  days.  As  a  result,  the  contractor  is  not  paid  until  three  months 
after  the  work  is  performed.  A  recent  contracting  officer  survey  reported  that  award-fees 
are  generally  assessed  at  80%  to  95%  of  the  allotted  award-fee  pool  (Snyder,  2001:13). 
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This  percentage  is  extremely  high  for  the  amount  of  work  and  effort  expended  during  the 
30  to  60  day  evaluation.  These  processes  are  often  successful,  but  at  what  expense? 

USAF  Acquisition  Reform 

This  study  attempts  to  work  within  the  constraints  of  the  Competition  in 
Contracting  Act  of  1984  while  attempting  to  push  the  boundaries  of  the  “transformation” 
movement  within  DoD.  “Transformation”  describes  the  harnessing  of  innovative 
organizational  changes  to  enhance  efficiency  and  effectiveness  (SECDEF,  2002). 
Transformation  in  the  USAF  acquisition  community  involves  strategic  sourcing.  An 
improved  supplier  selection  procedure  is  part  of  strategic  sourcing.  Transformation  is  a 
cultural  change  .  .  .  finding  people  who  think  right,  who  are  not  stuck  in  the  past  and  who 
are  willing  to  look  at  things  from  a  new  perspective  (SECDEF,  2002). 

Based  on  efforts  to  improve  USAF  procurement  processes,  the  office  of  the 
Undersecretary  of  the  Air  Force  for  Installation  and  Logistics  (AF/1L)  has  tailored  its 
own  version  of  the  PSCM  movement.  PSCM  consists  of  thirteen  key  tenets:  1)  align 
purchasing  strategies  with  enterprise  goals,  2)  gain  knowledge  of  where  the  enterprise’s 
money  is  spent,  3)  break  down  functional  stovepipes,  4)  focus  workforce  on  strategic 
objectives,  5)  select  sources  based  on  strategic  value,  6)  understand  supply  chain,  7) 
manage  key  suppliers,  8)  rationalize  supply  base,  9)  cultivate  long-term  supply  base,  10) 
automate  routine  activities,  11)  link  demand  planning  with  purchasing  strategies,  12) 
design  supply  chains  for  optimum  effectiveness,  and  13)  relentlessly  pursue  continuous 
improvement  (AF/1L  Briefing,  2002).  My  research  touches  on  areas  3-10.  Specifically, 
my  research  includes  supplier  qualifications  and  certifications  (Cook  &  Graser,  2001:92- 
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99).  In  general,  qualification  and  certification  involves  a  thorough  review  of  a  supplier’s 
internal  processes,  financial  records,  and  quality  control  standards. 

The  PSCM  movements  stem  from  the  “lean  aerospace  initiative”  which  proposes 
to  cut  the  costs  of  acquiring  future  systems  by  50%,  while  increasing  the  perfonnance 
and  quality  (Baker,  1998:22).  Optimizing  the  supplier  base  expedites  the  transformation 
process.  PSCM  attempts  to  improve  supplier  performance  (i.e.,  better  supplier  selection) 
by  maintaining  a  smaller  pool  of  high  quality  suppliers.  The  primary  motives  for 
reducing  the  number  of  suppliers  a  company  has  is  to  achieve:  1)  higher  quality,  2) 
ensure  timely  delivery,  and  3)  to  lower  purchasing  costs  (Piercy  and  Cravens,  1997:74). 
This  concept  is  linked  to  the  idea  of  strategic  sourcing,  which  is  the  corporate  decision  to 
integrate  purchasing  and  supply  personnel  into  teams  which  select  the  firms  most  capable 
of  meeting  the  government  requirement.  The  assumption  is  that  the  USAF  can  achieve 
this  by  implementing  commercial  practices. 

DoD  Efforts  and  Background 

Past  performance  information  is  relevant  information,  for  future  source 
selections  regarding  a  contractor’s  actions  under  previously  awarded  contracts.  It 
includes,  for  example,  the  contractor’s  record  of  conforming  to  contract 
requirements  and  to  standards  of  good  workmanship...  of  forecasting  and 
controlling  costs... [of]  adherence  to  contract  schedules,  including  the 
administrative  aspects  of  performance...  of  reasonable  and  cooperative  behavior 
and  commitment  to  customer  satisfaction;  and  generally,  the  contractor’s 
business-like  concern  for  the  interest  of  the  customer  (Federal  Acquisition 
Regulation  (FAR):  42.1501). 
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Pre- Award:  A  review  of  the  literature  indicates  that  NASA  was  the  first  government 
agency  to  implement  award-fee  contracts  and,  is  the  only  government  agency  currently 
using  supplier  ratings  for  pre-award  evaluation  (Snyder,  2001 : 15).  NASA  uses  a 
scorecard  rating  system  based  on  input  from  purchasing  personnel  to  predict  performance 
on  future  contracts. 

The  Air  Force  attempted  to  use  a  performance  rating  system  at  the  Air  Logistics 
Center  depots  in  1996.  It  was  known  as  the  Blue  Ribbon  Program  (BRP)  (Ambrose, 
1997:26).  It  was  a  best  value  award  system,  which  authorized  contracting  officers  to 
award  contracts  based  on  factors  deemed  more  important  than  price.  It  was  a 
certification  rating  designed  to  assess  performance  on  two  factors:  quality  and  timely 
delivery.  Contractors  were  designated  “blue  ribbon  certified”  and  their  credentials  for 
producing  designated  aircraft  parts  were  continuously  monitored.  Theoretically,  if  the 
BRP  firm  was  within  20%  of  the  lowest  bidder,  it  could  be  awarded  the  contract  based  on 
past  performance  (Ambrose,  1997:27).  The  program  failed.  The  Air  Force  cancelled  the 
program  because  it  required  excessive  documentation  for  contracting  officers  (Ambrose, 
1997:27).  The  strict  documentation  policy  was  necessary  to  comply  with  the  full-and- 
open  competition  requirement  of  the  CICA  (Ambrose,  1997:15).  CICA  was  established 
to  prevent  exclusion,  and  to  ensure  that  all  offerors  are  afforded  the  same  opportunity  to 
compete  for  business.  This  Act  is  worthwhile  in  principle,  but  troublesome  for 
acquisition  managers.  Contracting  Officers  are  often  inundated  with  paperwork  for 
award  fee  boards  and  past  performance  evaluations. 

The  Navy  also  implemented  a  program  known  as  the  Blue  Star  Program.  This 
program  was  also  based  on  quality  and  timely  delivery.  The  difference  between  this  and 
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the  BRP  was  that  it  did  not  give  preference  to  its  recipients.  Blue  Star  Contractors 
viewed  this  program  as  bragging  rights  among  their  competitors.  The  Navy  put  this 
program  on  hold  while  focusing  on  a  supplier  rating  system  using  a  Contractor  Report 
Card  (Ambrose,  1997:28). 

The  Army  instituted  a  similar  system  known  as  the  Army  Contractor  Performance 
Certification  Program  (CP2).  It  was  designed  to  emphasize  quality.  The  benefits  of  this 
program  were  difficult  to  assess,  although  it  did  allow  for  less  oversight  and  lower 
inspection  costs  (Ambrose  1997:29).  The  CP2  was  phased  out  because  of  the 
administrative  requirements  of  keeping  it  up  to  date. 

Current  guidance  on  past  performance  assessments  is  included  in  the  DoD  Past 

Perfonnance  Guide  (PPG)  of  2001  (Version  2).  This  document  outlines  the  performance 

areas  that  are  to  be  evaluated  during  pre-award. 

“The  past  perfonnance  factors  and  subfactors,  if  any,  should  be  designed 
to  evaluate  the  key  perfonnance  requirements  of  the  contract  solicitation. 

At  a  minimum,  the  solicitation  should  request  the  offeror's  record  for  on 
time  delivery,  technical  quality,  cost  control,  and  past  perfonnance  of 
subcontracting  plans/programs  ”  (PPG,  2001:10). 

The  categories  for  perfonnance  ratings  include:  Unsatisfactory/Very  High  Performance 

Risk,  Marginal/High  Performance  Risk,  Satisfactory/Moderate  Performance  Risk,  Very 

Good/Low  Performance  Risk,  Exccptional/V ery  Low  Performance  Risk,  and  Unknown 

Performance  Risk  (PPG,  2001 : 10). 

Post- A  ward:  Research  and  case-study  analysis  of  award-fee  contracts  date  back  to  the 
McNamara  days  of  the  1960’s.  While  award-fee  contracts  do  not  themselves  contain 
supplier  perfonnance  ratings,  they  do  include  factors  which  determine  the  monetary 
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incentive  offered  to  the  supplier  encouraging  the  supplier  to  meet  or  exceed  the  contract 
requirements  (Snyder,  2001:15). 

Current  guidance  for  USAF  post- award  evaluations  is  included  in  the  2001  Air 
Force  Materiel  Command  Award-Fee  Guide.  The  following  evaluation  areas  are 
included  as  a  sample. 


Figure  1.  AFMC  Award-Fee  Guide  (2000:83) 


Performance  Category 

COST  AND  SCHEDULE  MANAGEMENT 

COST  CONTROL 

COST  CONTROL/REPORTING 

COST  PERFORMANCE 

ORGANIZATION  AND  MANAGEMENT 

PROGRAM  MANAGEMENT 

QUALITY  ASSURANCE 

QUALITY  OF  WORK 

PRODUCT  QUALITY 

TECHNICAL  PERFORMANCE 

TIME  OF  DELIVERY 

SCHEDULE 

The  award-fee  board  members  assign  a  grade  of  Unsatisfactory,  Satisfactory,  Good,  Very 
Good,  or  Excellent  performance  after  evaluating  a  contractor  (AFMC  Award-fee  Guide, 
2000:81).  Each  category  is  weighted  in  establishing  a  grade  scale  of  0  to  100.  As 
mentioned  previously,  these  scores  are  generally  very  high  (80%  -  95%). 


10 


Air  Force  Audit  Findings 


The  Air  Force  Audit  Agency  conducted  an  audit  of  17  award-fee  contracts  in 
1991.  The  findings  included:  1)  award-fee  contract  types  lend  themselves  to  a  subjective 
as  opposed  to  an  objective  review;  2)  contracting  personnel  did  not  establish  an 
appropriate  methodology  for  assessing  performance  in  13  of  the  17  instances  (Snyder, 
2001 : 17).  The  opinion  that  supplier  perfonnance  ratings  in  civilian  industry  are 
subjective  does  not  differ  from  the  findings  of  the  AFAA.  The  industry  applications 
encountered  below,  include  objective,  as  well  as  subjective  ratings. 

Industry  Solution 

Industry  has  struggled  with  supplier  management  just  as  the  USAF  has.  During 
the  production  of  the  Boeing  747  and  737  airplanes,  Boeing  supplier  failure  cost  over 
$1B  (Park  et  ah,  2001:696).  To  solve  problems  like  these,  many  commercial  firms  have 
devised  a  paperless  media  for  tracking  their  suppliers.  Supplier  rating  systems  are 
growing  in  popularity  in  the  civilian  marketplace.  Many  commercial  firms  use  software 
driven  performance  ratings  that  combine  qualitative  and  quantitative  measures.  One 
example  of  the  software  driven  system  is  Open  Ratings  (OR).  OR  gathers  opinion  data, 
transaction  data,  and  third-party  financial  data  information  from  Dun  &  Bradstreet  in 
computing  the  overall  score  (OpenRatings.com,  2002).  More  specifically,  the  OR  system 
measures  overall  performance  based  on  reliability,  cost,  order  accuracy,  delivery- 
timeliness,  quality,  business  relations,  personnel,  customer  support,  and  responsiveness. 
Supply  chain  managers  use  the  database  infonnation  to  choose  between  multiple 
suppliers.  In  effect,  the  software  is  now  the  predictor  for  determining  the  outcome  of  a 
contract  in  a  quantifiable  manner  (Smith,  2000:40).  By  referring  to  the  real-time  ratings, 
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companies  can  predict  good  vs.  bad  partners.  This  system  allows  the  buyer  to  shorten  the 
pre-award  evaluation  and  to  substitute  award-fee  boards  with  percentages  awarded 
according  to  the  Standard  Industrial  Code  (SIC).  The  SIC  code  informs  the  buyer  what 
type  of  work  a  supplier  is  capable  of  performing.  A  sample  report  from  Open  Ratings 
(Attachment  1)  is  included.  I  have  also  included  a  scorecard  from  UTC  which  shows  the 
ratings  for  a  particular  firm  across  each  division  of  UTC  (Attachment  2). 

FedEx  also  has  a  software  generated  system  in-place.  FedEx’s  system  was 
chosen  as  they  are  a  leader  in  the  airborne  logistics  industry.  I  conducted  an  interview 
with  one  of  FedEx’s  strategic  sourcing  managers,  Joseph  K.  Clark,  to  discuss  FedEx’s 
supplier  management  program.  FedEx’s  evaluation  categories  include  strategy, 
resources,  process,  optimization,  and  globalization  (Clark).  Their  system  was  built  by 
purchasing  personnel  at  corporate  headquarters  in  Memphis,  TN.  FedEx’s  management 
assigns  accountability  to  purchasing  and  logistics  personnel  when  selecting  suppliers. 
Using  accountability  in  purchasing  terms,  it  represents  purchasing’s  responsibility  to 
achieve  pre-determined  goals  and  attaining  perfonnance  thresholds  (Zsidisin  and  Ellram, 
2001:631). 

The  database  at  FedEx  addresses  past  perfonnance  evaluations  and  post-award 
considerations  as  well.  During  contract  generation,  a  clause  explains  the  balanced 
scorecard  system  to  the  vendor.  There  are  five  categories  at  FedEx:  Unacceptable, 
Bronze,  Silver,  Gold,  and  Platinum.  For  instance,  if  the  vendor  agrees  to  a  3%  award-fee 
for  subsequent  years,  they  will  get  that  with  a  silver  rating.  If  they  achieve  a  gold  rating, 
they  receive  a  3.5%  award-fee  during  that  year.  If  they  remain  at  bronze  or  below,  they 
will  receive  2.5%  or  less  (Clark). 
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Buyer/Supplier  Relations 


Supplier  selection  is  the  primary  task  for  purchasing  managers  (Monczka  et  al., 
2002:222-265).  This  task  is  becoming  more  complex  in  the  global  business  environment 
today  because  selection  decisions  are  now  strategic  and  include  analysis  based  on  total 
cost  of  ownership  (TOC)  (Ellram,  1995:4-23).  TOC  includes  every  aspect  associated 
with  the  life-cycle  of  the  product.  How  much  will  it  cost?  How  much  will  it  cost  to 
maintain?  Is  the  system/product  reliable? 

The  traditional  criteria  of  price,  quality,  delivery,  and  speed  are  changing  to 
include  financial  data.  Due  to  proprietary  information  and  traditional  thinking,  this  was 
extremely  uncommon  between  firms.  Figure  2  summarizes  the  changes  in  buyer/supplier 
relations. 
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Figure  2.  Source  (Goffin  et  al.,  1997:422-436) 
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Principal  Component  Analysis  (PCA)  &  Data  Envelopment  Analysis  (PEA) 

Recent  studies  in  2000  and  2001  were  conducted  to  identify  ways  for  purchasing 
managers  to  identity  the  factors  which  are  most  important  when  selecting  suppliers 
(Narasimhan  et.  al.,  2001  and  Petroni  and  Braglia,  2000).  The  results  of  the  Petroni  and 
Braglia  PCA  study  produced  a  formula  which  measures  three  outputs  (performance 
measurements)  and  three  inputs  (supplier  attributes)  using  the  traditional  indicators  of 
product  price,  shipment  quality,  and  delivery  compliance  (Petroni  and  Braglia,  2000:64- 
65).  The  results  of  the  PCA  for  each  indicator  are  computed  on  a  scale  from  .001  to  1 
and  summed  to  rank  suppliers. 
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The  Narasimhan  et  al.  DEA  study  applies  non-parametric  analysis  models  to 
predict  those  factors  which  are  most  important  in  supplier  selection.  The  deemed  six 
categories  most  important:  1)  quality  management  practices  and  systems,  2) 
documentation  and  self-audit,  3)  process/manufacturing  capability,  4)  management  of  the 
firm,  5)  design  and  development  capabilities,  6)  cost  reduction  capability  (Narasimhan  et. 
al.,  2001 :3 1).  The  researchers  used  these  categories  and  sent  questionnaires  to  Company 
X’s  purchasing  employees  to  rate  the  capability  of  their  suppliers. 

Channel  Equity 

A  recent  study  indicated  that  less  than  half  of  the  firms  in  civilian  industry  have  a 
formal  supplier  evaluation  in  place  (Simpson  et.  al.,  2002:29).  This  study  suggests  that 
firms  must  strive  to  achieve  channel  equity.  Channel  equity  is  the  successful  attempt  to 
identify  suppliers  that  provide  the  greatest  business  synergy  through  a  fonnalized 
evaluative  process  (Simpson  et.  al.,  2002:30).  In  doing  this,  it  is  critical  that  a  firm 
identify  factors  other  than  price,  quality,  and  delivery.  It  is  now  more  important  than  ever 
that  business  partners  establish  a  win-win  relationship  with  its  customers.  The  surveys  of 
the  Simpson  et.  al.  study,  which  were  given  to  1 10  of  the  top  150  Fortune  companies, 
identified  customer  relationships  and  communication  as  the  most  critical  aspect  of 
supplier  evaluations.  As  such,  my  thesis  will  address  customer  relationships  and 
communication  in  terms  of  pre-award  and  post-award  evaluations. 

Scope 

The  commercial  scope  of  this  effort  includes  contractors  that  are  in  the  “Fortune 
500  Top  5”  of  either  the  government  or  air  transportation  industry.  The  government 
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scope  if  limited  to  Aeronautical  Systems  Center  (ASC)  at  Wright-Patterson  AFB,  OH. 
Further,  the  focus  of  my  research  emphasizes  assessments  which  are  based  on  source 
selections  and  award- fee  boards  which  were  perfonned  at  ASC. 

Conclusion 

This  chapter  summarizes  the  push  for  refonn  in  the  acquisition  community  and 
the  implementation  of  commercial  practices.  The  discussion  covers  DoD  attempts  to 
devise  supplier  selection  methods  and  industry  examples  of  how  they  are  reducing  the 
number  of  suppliers  they  use.  An  overview  of  supplier  criteria  and  the  change  in  the 
buyer/supplier  interchange  was  also  provided.  Chapter  3  discuses  the  methodology  used 
to  collect  and  analyze  the  available  data. 
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III.  Methodology 


The  purpose  of  this  chapter  is  to  describe  the  research  objectives  that  the 
researcher  must  meet  in  order  to  complete  this  study  and  the  methods  employed  to  meet 
those  objectives.  This  chapter  begins  with  an  explanation  of  what  case  study  research  is, 
and  why  I  have  chosen  it  to  address  my  research  questions.  Additionally,  this  chapter 
discusses  the  research  design  as  it  applies  to  the  gathering  of  data.  Finally,  it  addresses 
the  issues  of  validity  and  reliability. 

Research  Objectives 

The  objective  of  this  research  is  to  assess  supplier  evaluation  methods.  This 
inductive  study  includes  an  investigation  of  pre-award  and  post-award  evaluations 
conducted  by  civilian  companies  and  USAF  program  offices  alike.  In  doing  so,  I  hope  to 
uncover  patterns  for  success  that  civilian  companies  have  used  to  expedite  the  process 
while  maintaining  the  integrity  of  the  process.  If  acquisition  reform  is  about  getting  the 
products  or  systems  to  the  warfighter  more  rapidly,  then  speeding  up  pre-award  and  post¬ 
award  assessments  are  steps  in  the  right  direction. 

Method 

This  study  employs  a  case  study  methodology.  This  type  of  analysis  requires  an 
extensive  analysis  of  three  conditions:  the  type  of  research  question  posed,  the  extent  of 
control  the  investigator  has  over  actual  behavioral  events,  and  the  degree  of  focus  on 
contemporary  as  opposed  to  historical  events  (Yin,  1994:4).  The  use  of  past  performance 
and  award  fees  are  contemporary  as  opposed  to  historical  events  (Yin,  1994:4).  Each 
case  under  study  is  less  than  3  years  old. 
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I  chose  this  method  for  several  reasons.  First,  the  case  study  allows  us  to  compare 
the  data  that  the  USAF  currently  collects  on  its  suppliers  with  that  of  the  data  that 
commercial  companies  use  to  rate  their  suppliers.  Second,  we  have  no  foundational  data 
available  on  the  relationship  between  supplier  rating  systems  and  USAF  contracts.  Third, 
USAF  acquisition  is  not  much  different  than  that  of  some  commercial  companies.  DoD 
acquisition  reforms  attempt  to  use  innovative  practices  that  are  found  in  commercial 
companies.  The  research  of  this  study  is  defined  as  exploratory.  The  goal  is  to  build 
theory  rather  than  test  it.  The  choice  of  case  study  adds  two  sources  of  evidence  to  the 
research’s  choices,  direct  observation  and  systematic  interviewing  (Yin,  1994:12).  Case 
studies  can  include  either  single-  or  multiple  case  studies.  This  research  uses  the 
multiple-case  study  approach,  also  known  as  the  comparative  case  method.  It  will 
include  quantitative  data  as  well  as  qualitative.  The  qualitative  data  that  was  obtained 
included:  business  strategy  (price  trends),  resources  (services),  processes  (discrepancy 
rate,  financial  stability,  and  certifications),  optimization  (cycle  time  performance)  and 
globalization  (coverage,  compatibility).  The  quantitative  data  include:  time  required  to 
complete  evaluation  and  delivery  time. 

To  complete  the  study,  I  will  conduct  phone  interviews  and  personal  interviews 
and  query  several  points  of  interest.  These  include:  evaluation  factors,  automated  vs. 
manual  evaluation  methods,  timelines  for  supplier  evaluation,  DoD  acquisition 
restrictions  for  evaluation,  and  supplier  relationships. 

Selecting  an  Appropriate  Case  to  Study 

Before  determining  which  cases  were  appropriate  to  study,  I  first  examined  the 
population  of  interest.  In  this  instance,  the  population  is  all  commercial  companies  and 
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USAF  acquisition  programs.  Our  sampling  frame  begins  with  Fortune  500  companies 
that  participate  in  DoD  acquisition  or  currently  work  in  the  aerospace  industry  (e.g. 
aircraft  manufacturing,  parts  suppliers,  air  transportation).  Some  of  the  companies 
include  Federal  Express,  United  Technologies,  Delta  Airlines,  and  Lockheed  Martin. 
These  companies  conduct  pre-award  and  post-award  evaluations  on  their  suppliers.  On 
the  USAF  side,  I  chose  two  source  selection  programs  at  Wright-Patterson  Air  Force 
Base,  OH.  They  included  the  C-130  Avionics  Modernization  Program  (C-130  AMP)  and 
the  Air  Force  Research  Laboratory  Enterprise  Business  System  (AFRL  EBS).  The 
individuals  from  the  C-130  AMP  and  the  AFRL  EBS  program  had  experience  with  both 
pre-award  and  post-award  evaluations.  Here  I  conducted  interviews  with  the  program 
manager  or  contracting  officer  who  were  involved  with  a  source  selection  or  award-fee 
boards.  The  assumption  is  that  the  C-130  AMP  and  the  AFRL  EBS  are  good 
representatives  of  USAF  acquisition  programs.  The  C-130  AMP  is  a  large  program 
valued  at  $2B  to  $3B  and  the  AFRL  EBS  is  a  small  program  valued  at  $15M  program. 

Interview  Process 

The  civilian  contractor  personnel  were  interviewed  to  detennine  what  quantitative 
and  qualitative  data  is  collected  in  order  to  establish  their  supplier  ratings.  In  addition, 
they  were  asked  to  evaluate  the  effectiveness  of  their  rating  systems  and  how  they  were 
constructed.  The  government  personnel  were  then  interviewed  to  detennine  how  data  is 
collected  and  whether  it  would  be  feasible  to  utilize  a  more  automated  system  for  USAF 
acquisitions.  The  interview  process  also  included  a  question  regarding  time  required  to 
complete  the  award  fee  board.  The  time  required  to  decide  on  award  fee  percentages  was 
very  important  in  detennining  whether  the  supplier  rating  system  was  useful.  After  all,  if 
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time  was  not  saved  then  it  becomes  redundant  to  create  supplier  ratings.  Chapter  4 
addresses  the  government  time  requirements  as  opposed  to  the  time  it  takes  the  contractor 
to  maintain  its  supplier  ratings. 

There  are  three  types  of  interviews:  infonnal  conversation,  interview  guide,  and 

the  standardized  open-ended  interview  (Patton,  1990).  The  informal  interview  is 

conducted  through  nonnal  conversation.  This  approach  is  popular  because  of  its 

flexibility.  Our  interview  combined  the  informal  conversation  and  the  interview  guide 

approach.  Through  the  use  of  telephone  interviews,  I  asked  commercial  company 

employees  a  series  of  standardized  questions.  I  used  a  second  set  of  questions  which 

were  similar  to  the  commercial  company  questions,  for  the  government  program 

personnel.  The  USAF  program  office  interviews  were  conducted  via  face-to-face 

interviews.  This  was  primarily  due  to  the  proximity  of  my  location  and  the  program 

offices,  but  also  due  to  the  complexity  of  the  source  selection  and  award-fee  evaluations. 

To  ensure  each  contact  was  informed  of  the  same  information,  the  researcher  used  an 

initial  telephone  contact  guide.  A  set  of  basic  questions  was  addressed  during  the 

interviews  and  while  reviewing  the  literature  and  documentation.  The  questions 

addressed  were  open-ended  and  dynamic  in  order  to  facilitate  rich  discourse.  The 

interview  questions  were  initially  mapped  to  the  study’s  research  questions  as  follows. 

Research  Question  1 :  Can  USAF  acquisition  personnel  use  supplier  ratings  in  lieu  of 
conducting  award-fee  boards  or  past  performance  evaluations? 

Interview  Question  (IQ)  10  .  How  is  post-award  evaluation  maintained  by  your 
firm/program  office? 

IQ  1 1  Is  the  process  automated  or  manual?  What  is  the  role  of  purchasing  personnel  in 
this  process? 
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IQ  4C.  (C=Contractor  Only)  Do  you  maintain  one  rating  for  each  supplier  which  covers 
all  contracts  or  are  they  given  a  rating  which  is  based  on  their  performance  on  one 
particular  contract? 

IQ  16.  What  3  things  could  you  have  done  better  in  pre-award  and  post-award 
evaluations  knowing  what  you  know  now? 

3G.  (G=Govemment  Only)  If  this  process  could  be  automated  would  it  alter  your 
perception  of  the  DoD  process? 

Investigative  Question  2:  What  are  the  similarities  between  USAF  rating  categories  and 
commercial  rating  categories?  (These  are  the  factors  which  are  used  in  award-fee 
determinations  and  past  performance  evaluations) 

IQ  1.  What  are  the  categories/factors  of  assessment  used  for  evaluating  past 
performance?  How  were  these  categories  constructed  (i.e.  industry  standard,  from 
within)? 

IQ  2.  Is  your  rating  system  quantitative  (e.g.  late  delivery  in  terms  of  #  of  days)  or 
qualitative  (e.g.  quality  performance)? 

IQ  3.  How  many  days  on  average  does  it  take  you  to  award  a  $50M  contract  (e.g.  clock 
starts  when  a  requirement  or  funding  document  is  received)? 

IQ  4.  Has  your  pre-award  evaluation  process  changed  over  the  last  5  years 

IQ  7.  Would  you  categorize  most  of  your  contracts  fixed-price,  incentive  based,  or  cost- 
plus? 

IQ  8.  Is  the  importance  of  price  or  financial  performance  tailored  for  each  acquisition? 

IQ  9.  Is  price  the  most  important  factor  in  evaluation? 

IQ  17.  How  much  training  is  given  in-house  regarding  supplier  evaluations? 

IQ  18.  Does  your  firm  use  multi-functional  teams  when  perfonning  pre-award  and  post¬ 
award  evaluations  or  do  different  departments  (e.g.  engineering,  purchasing,  marketing) 
act  separately? 

Investigative  Question  3:  Can  cost-avoidance  savings  be  quantified  if  either  were 
replaced? 

IQ  3.  How  many  days  on  average  does  it  take  you  to  award  a  $50M  contract? 
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IQ  5.  Have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your 
evaluation  process? 

IQ  12a.  How  much  time  (hours,  days)  is  required  to  complete  a  post-award  evaluation? 

IQ  12b.  How  often  is  it  perfonned  (never,  monthly,  quarterly)? 

IQ  19.  How  many  individuals  typically  participate  in  a  pre-award  or  post-award 
evaluation  (i.e.  $1M,  $50M,  &  $100M+  acquisitions)? 

Investigative  Question  4:  What  acquisition  reform  initiatives  or  regulations  would  aid  or 
hinder  USAF  application  of  a  revised  performance  evaluation  system? 

IQ  1G.  What  is  your  perception  of  the  FAR’s  restrictions  on  the  pre-award  evaluation 
process? 

IQ  1C.  What  is  your  impression  of  the  current  DoD  past  performance  and  risk 
assessment? 

IQ  6C.  (Contractor  Only)  What  regulations  does  your  company  have  which  outline  the 
pre-award  evaluation  process? 

Pattern  Matching 

Yin  suggests  a  “pattern-matching”  approach,  first  described  by  Donald  Campbell 
in  1975  (Yin,  1994:25).  Pattern  matching  is  a  process  whereby  bits  of  information  and 
findings  from  each  case  are  “matched”  to  a  theoretical  proposition.  In  this  case,  I  used 
the  matching  technique  to  relate  infonnation  from  each  individual  case  to  the  questions 
and  objectives  described  in  Chapter  1.  Accordingly,  by  reviewing  these  relationships  and 
analyzing  their  importance,  I  was  able  to  fonnulate  conclusions  and  recommendations 
found  in  Chapter  5. 

Case  Study  Reliability  &  Validity 

The  objective  of  this  study  was  to  establish  the  reliability  and  validity  of  supplier 
rating  methods.  The  questions  used  to  interview  the  contractor  personnel  and 
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government  personnel  were  submitted  for  a  consensus  from  the  research  committee  prior 
to  execution. 

Content  validity  is  not  detennined  using  statistical  techniques;  instead,  it  is 
determined  through  a  review  of  literature  and  review  of  previous  research  in  the  area 
being  studied  (Emory,  1980).  An  expert  in  the  area  of  research  methods  was  consulted 
and  agreed  that  the  interview  questions  appeared  effective  in  determining  what  factors  are 
included  in  supplier  ratings  and  what  factors  are  used  by  government  evaluators  for 
award-fee  or  past  perfonnance.  The  interviews  were  transcribed  entirely  and  analyzed 
using  open-ended  content  analysis.  The  software  tool  “StatPac”  was  using  to  eliminate 
any  bias  during  content  analysis.  Essentially,  StatPac  performed  the  analysis  by 
summing  the  frequency  of  key  tenns  appearing  in  each  question  of  each  interview  for  all 
10  cases  (8  commercial  companies  and  2  government  offices).  To  ensure  the  accuracy  of 
StatPac’s  reports,  which  are  included  in  Chapter  4,  a  visual  count  was  done  to  eliminate 
duplicate  words  within  the  same  dialogue  of  one  individual  interview.  Additionally, 
words  were  grouped  to  capture  similar  terms. 

Threats  to  Validity 

Good  research  design  maximizes  external  validity,  reliability,  construct  validity, 
and  internal  validity  (Ellram,  1996:104).  “External  validity  reflects  how  accurately  the 
results  represent  the  phenomenon  studied”  (Ellram,  1996: 104).  External  validity  is 
threatened  when  there  is  not  a  causal  relationship  between  the  constructs  of  the  cause  and 
the  effect.  In  my  case,  if  organizations  use  automated  supplier  rating  systems  and  I  can 
not  show  that  that  it  improves  their  overall  business  perfonnance  then  external  validity  is 
threatened.  Reliability  is  threatened  when  it  is  not  possible  to  achieve  the  same  results 
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when  repeating  the  experiment  (Ellram,  1996:104).  In  my  case,  I  used  case  study 
protocol  in  creating  the  interview  questions  and  these  questions  were  corroborated  with 
committee  members  (Ellram,  1996:104-105).  Construct  validity  “addresses 
establishment  of  the  proper  operational  measures  for  the  concepts  being  studied”  (Ellram, 
1996:105).  To  eliminate  this  threat,  I  used  multiple  sources  of  evidence,  established  a 
chain  of  events,  and  had  key  informants  review  the  case  study  research  (Ellram, 
1996:105).  In  my  case,  I  used  phone  and  personal  interviews  and  I  also  used  information 
submitted  by  commercial  companies.  Three  committee  members  examined  the  entire 
document  and  reviewed  the  research  to  ensure  that  there  was  a  logical  flow  (Ellram, 
1996:106).  Construct  validity  is  threatened  when  assumptions  are  made  regarding 
measures  and  how  they  reflect  constructs.  Are  the  constructs  reflected  by  my  interview 
questions?  For  example,  question  19  asks  how  many  personnel  participate  in  pre-award 
and  post-award  evaluations.  I  am  assuming  that  this  question  is  related  to  how  large  the 
program  is  in  tenns  of  total  dollars.  This  may  not  be  related  to  overall  program  cost. 
Internal  validity  is  threatened  when  an  assumption  is  made  that  the  relationship  is  a 
causal  one.  In  my  study,  I  am  assuming  that  the  supplier  evaluation  practices  of 
commercial  companies  allow  them  to  expedite  the  buying  process.  Internal  validity 
would  be  threatened  if  automated  supplier  ratings  were  not  linked  to  the  time  required  to 
select  a  new  supplier. 

Validity  types  build  on  one  another.  It  is  the  goal  of  my  research  to  build  a  bridge 
from  each  validity  type.  The  effect  is  cumulative.  Attempts  are  made  to  minimize 
validity  threats  in  sequence. 
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Generalizing/Summary 


Although  this  research  focuses  on  USAF  acquisition  it  can  be  applied  to  DoD 
acquisition.  It  also  assumes  that  the  “Purchasing  and  Supply  Chain  Management” 
(PSCM)  movement  is  an  innovative  way  of  doing  business.  The  PSCM  concept  is 
heavily  focused  on  supplier  development.  Supplier  ratings  are  one  means  to  improve 
supplier  development. 
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IV.  Results 


This  chapter  analyzes  the  3 1  questions  that  were  included  in  the  interviews  with 
commercial  company  personnel  and  government  program  office  personnel.  Each 
question  includes  the  StatPac  frequency  analysis  data  which  identifies  the  emerging 
patterns  and  results  from  the  case  study  interviews.  There  are  also  specific  examples 
listed  to  provide  rationale  for  findings.  Questions  1-21  were  asked  of  both  commercial 
and  government  organizations.  Questions  1C-6C  were  asked  of  commercial  companies 
only  and  1G-4G  were  asked  of  government  only. 

Note:  The  percentages  are  a  proportion  of  a  particular  response  with  that  of  how  many 
organizations  were  interviewed.  In  questions  1-21  the  “n”  is  10.  In  questions  1C-6C  the 
“n”  is  8,  and  in  questions  1G-4G  the  “n”  is  2.  It  was  possible  for  a  company  to  list  more 
than  one  response.  For  Example:  In  Question  1,  there  were  7  comments  regarding  on- 
time  delivery,  and  if  7  is  divided  by  10  it  can  be  interpreted  that  70%  of  the  organizations 
said  that  on-time  delivery  is  important. 


Pre- Award  Evaluation 


Question  la  “What  are  the  categories/factors  of  assessment  used  for  evaluating  past 


performance?” 

_ Number _ Percent 

on-time  delivery  7  70.0  % 

overall  performance  6  60.0  % 

quality  6  60.0  % 

price/cost  5  50.0  % 

technical/engineering  4  40.0  % 

financial  health  4  40.0  % 

program  management _ 3 _ 30.0  % 


n=  10 
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This  question  was  constructed  to  provide  rationale  for  what  data  should  be 
collected  assuming  a  company  or  organization  is  interested  in  establishing  a  formal 
supplier  rating  system.  The  most  common  factor  mentioned  was  on-time  delivery  with 
seven  out  of  10  reporting  its  importance.  Quality  and  past  performance  were  also 
important.  Some  companies  mentioned  price  or  cost,  and  although  they  are  not  the  same, 
the  terms  were  joined  for  simplification.  In  the  two  government  cases,  the  source 
selection  teams  used  Contractor  Perfonnance  Assessment  Reports  (CPARs)  criteria  or 
previous  source  selection  factors  to  establish  their  criteria.  The  CPARs  included 
information  on  product  perfonnance,  systems  engineering,  software  engineering,  logistics 
support/sustainment,  product  assurance,  schedule,  cost  control,  management, 
management  responsiveness,  subcontract  management,  and  program  management.  The 
Air  Force  Research  Laboratory  Business  Enterprise  System  evaluated  management, 
technical,  functionality,  operations  and  sustainment,  and  cost  perfonnance  (AFRL 
Interview).  Raytheon  reported  that  they  look  at  on-time  delivery  internally,  quality  in 
number  of  lots  received  vs.  rejected,  and  benchmarks  with  other  competitors  (Raytheon 
Interview).  Boeing  reported  that  they  look  at  quality,  acceptance  rate,  on-time  delivery, 
and  customer  satisfaction  (Boeing  Interview).  United  Airlines  pointed  out  that  there  is  a 
huge  difference  when  dealing  with  new  suppliers.  “If  I  choose  a  new  company  I  want  to 
know  if  they’re  financially  viable,  if  they  can  deliver  on  time,  how  close  they  are  located 
to  our  warehouse,  and  how  they  perform  technically”  (United  Interview).  United 
Technologies  Corporation  (UTC)  reported  8  categories  for  assessment.  They  look  at 
delivery,  quality,  financial  perfonnance,  lean  manufacturing  initiatives,  manufacturing 
capabilities,  cost  competitiveness,  and  OpenRatings  (UTC  interview). 
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Question  lb  “How  were  these  categories  constructed  (i.e.  industry  standard,  from 
within)?” 


from  within 
industry  standard 


Number 

7 

3 


Percent 
70.0  % 
30.0  % 


n  =  10 

Companies  were  asked  whether  they  had  constructed  their  rating  systems  on  their 

own  or  based  on  an  industry  standard.  StatPac  excluded  the  word  “within”  from  the 

analysis.  In  counting  the  responses,  I  discovered  that  5  of  the  8  commercial  companies 

had  established  their  factors  on  their  own.  The  other  3  were  established  using  industry 

standards.  The  USAF  offices  were  also  grouped  in  the  “within”  category. 

Question  2  “Is  your  rating  system  quantitative  (e.g.  late  delivery  in  terms  of  #  of  days)  or 
qualitative  (e.g.  quality  performance)?” 


qualitative  only 
uantitative  only 


Number 

7 

2 

1 


Percent 
70.0  % 
20.0  % 
10.0  % 


The  most  common  quantitative  measure  included  was  on-time  delivery.  In  many 
cases,  this  data  was  retrieved  automatically.  This  question  produced  an  interesting 
difference  between  commercial  companies  and  government  organizations.  The 
commercial  companies  expressed  a  desire  to  make  their  rating  categories  as  quantitative 
as  possible.  However,  the  commercial  companies  explained  that  a  mixture  of  the  two 
was  more  common.  On  the  contrary,  the  government  organizations  explained  the 
importance  of  qualitative  (subjective)  ratings.  Delta  Airlines  said  they  go  to  great  lengths 
to  make  all  their  rating  categories  as  quantitative  as  possible  (Delta  Interview). 

Honeywell  reported  that  they  use  a  mix  of  quantitative  and  qualitative  measures.  “When 


28 


dealing  with  different  industries  you  expect  them  to  adhere  to  different  quality  standards 
(e.g.  casting/forging  vs.  simple  parts),  as  such  it’s  hard  to  be  all  quantitative  with  quality” 
(Honeywell  interview).  Raytheon  explained  that  they  try  to  be  objective  but  that  there 
are  times  when  they  must  be  subjective  (Raytheon  Interview).  Raytheon  also  provided  a 
measure  of  quantitative  measure  of  quality  as  number  of  lots  vs.  number  of  items 
rejected.  Boeing  stated  that  their  rating  system  was  quantitative  in  tenns  of  quality, 
acceptance  rate,  on-time  delivery  and  qualitative  in  tenns  of  customer  satisfaction.  UTC 
also  uses  a  mix.  UTC  listed  their  quantitative  measures  as  delivery,  quality,  and 
financial/cost  competitiveness  and  their  qualitative  measures  as  lean  manufacturing 
initiatives,  engineering  initiatives,  manufacturing  capabilities,  and  manufacturing 
capacity  (UTC  interview).  Lockheed  Martin  assesses  quality  and  delivery  quantitatively 
and  affordability  and  management  as  qualitative  (LM  Interview).  “We  e-mail  quality  and 
delivery  information  to  our  suppliers  on  a  monthly  basis.  The  qualitative  infonnation  is 
sent  semi-annually”  (LM  Interview).  The  government  C-130  Avionics  Modernization 
Program  said  their  source  selection  was  extremely  qualitative.  “We  made  every  effort  to 
remain  subjective,  because  objective  ratings  are  hard  to  define  in  the  request  for 
proposals  and  this  was  common  to  most  source  selections  and  SAF/AQC  guidance”  (C- 
130  AMP  Interview). 
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Question  3  “How  many  days  on  average  does  it  take  you  to  award  a  $50M  contract  (e.g. 
clock  starts  when  a  requirement  or  funding  document  is  received)?” 


_ Number _ Percent 

1-3  months 

it  depends  on  complexity 

1  year _ 

n  =  10 

This  question  was  difficult  to  evaluate  across  interviewees.  All  of  those 
interviewed  asked  for  a  dollar  value  in  order  to  answer  the  question.  The  example  of  a 
$50M  buy  was  provided.  Answers  ranged  from  3  weeks  to  one  year.  Overall,  this 
answer  was  dependent  upon  the  urgency  of  the  requirement.  Honeywell,  gave  an 
example  of  evaluating  a  supplier  in  Mexico.  In  the  global  environment,  “we  have  to  be 
very  confident  that  the  supplier  really  understands  the  terms  and  conditions,  so  this 
process  may  take  6  months”  (Honeywell  interview).  Most  explained  that  this  process  his 
highly  dependent  on  the  complexity  of  the  buy.  United  Airlines  said  that  a 
telecommunications  bid  for  $50M  might  take  8  months,  while  a  large  buy  for  plastics 
products  could  be  completed  by  reverse  auctioning  in  one  afternoon  (United  Airlines 
interview).  The  C-130  AMP  program  was  not  relative  to  the  $50M  example.  This  buy 
was  $2B  to  $3B  and  took  more  than  a  year.  The  AFRL  EBS  program  was  $15M  and  still 
took  almost  a  year.  Commercial  companies  appear  to  be  able  to  evaluate  and  select 
suppliers  more  quickly  because  of  a  more  automated  and  quantitative  assessment. 
However,  there  is  not  enough  infonnation  to  test  for  this. 


6  42.9  % 

3  21.4% 

2  14.3  % 
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Question  4  “Has  your  pre-award  evaluation  process  changed  over  the  last  5  years?” 


yes,  significantly 
not,  sign i  He iantly 
rating  system  changed 
more  automated 


Number _ Percent 

5  50.0  % 

4  40.0  % 

3  30.0  % 

2  20.0  % 


n  =  10 


Most  companies  stated  their  process  had  become  more  automated.  Delta  Airlines 
stated  that  they  process  has  changed  in  the  way  they  collect  and  distribute  information  to 
suppliers  (Delta  Airlines).  Companies  such  as  United,  Honeywell,  and  FedEx  are  doing 
reverse  auctions  with  prc-qualilied  vendors.  Raytheon  stated  that  they  have  changed 
from  a  tool  and  process  standpoint .  .  .  we’ve  gotten  more  electronic  and  web-enabled” 
(Raytheon  Interview).  The  government  personnel  stated  that  most  pre-award  evaluations 
are  conducted  in  the  same  manner.  The  source  selection  process  for  a  Performance  Risk 
Assessment  Group  (PRAG)  is  more  structured  but  we  are  evaluating  the  same  supplier 
performance  categories  (AFRL  interview).  Overall,  civilian  companies  appear  to  have 
made  a  more  profound  attempt  to  automate  this  process. 

Question  5  “Have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your 


evaluation  process?” 

_ Number _ Percent 

yes,  savings  (e.g.  delivery  time,  quality)  4  40.0  % 

dollar  savings  2  20.0  % 

tried  to,  but  no  results _ 5 _ 50.0  % 


n  =  10 

Many  of  the  answers  for  this  question  were  not  in  put  tenns  of  dollars;  however, 
they  are  mentioned  because  they  appear  to  be  important.  Honeywell  explained  that 
through  increased  supplier  stratification  they  have  been  able  to  shrink  the  supply  base  and 
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decrease  defects,  thus  providing  for  a  25%  improvement  in  the  quality  level  (Honeywell 
Interview).  UTC  reported  to  Wall  Street  a  $1B  savings  through  2001  on  the  product  side 
and  a  $700M  savings  on  the  non-product  side  by  the  year  2004  (UTC  Interview).  This 
was  due  largely  to  their  improved  sourcing  procedures.  Delta  Airlines  listed  a  10% 
savings  due  to  improved  on-time  delivery  over  the  last  three  years  (Delta  Airlines 
Interview).  FedEx  said  they  have  achieved  a  5%  to  10%  savings  using  reverse  auctions 
with  E-sourcing  (FedEx  Interview). 

Question  6  “Does  your  company  receive  feedback  from  other  business  partners 
concerning  your  own  past  perfonnance  evaluation  (e.g.  how  you  are  perceived  or  rate 


with  other  firms)?” 

_ Number _ Percent 

yes  8  80.0  % 

briefings/conference/debrief  3  30.0  % 

electronic  feedback  3  30.0  % 

phone  complaints _ 3 _ 30.0  % 


n  =  10 

Eight  of  the  ten  cases  said  that  they  receive  feedback  from  suppliers.  In  the 
Government’s  case  the  feedback  came  during  the  debriefings  at  the  completion  of  the 
source  selections  (C-130  AMP  Interview  &  AFRL  Interview).  Often,  the  government 
will  de-brief  the  contractors  concerning  their  past  performance  ratings  so  that  there  are  no 
surprises  in  the  end.  The  feedback  that  potential  suppliers  receive  during  government  de¬ 
briefs  is  usually  conducted  in  a  fonnal  manner.  In  the  commercial  sector  this  feedback 
occurs  through  both  formal  and  informal  feedback  mechanisms.  “We  get  feedback 
informally,  because  we  don’t  currently  have  a  fonnal  mechanism  for  doing  that”  (Delta 
Interview).  “Our  suppliers  are  trained  on  how  we  are  going  to  measure  them  and  it  is 
often  more  complicated  than  you  think  it  would  be”  (Delta  Interview).  Honeywell  stated 
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that  they  receive  most  of  the  feedback  during  FAA  audits.  “One  of  the  biggest 
complaints  is  that  the  original  equipment  manufacturers  (OEMs)  are  not  managing  their 
suppliers  very  well,  and  Boeing  took  us  to  task”  (Honeywell  Interview).  “When  Boeing 
comes  to  Honeywell  and  asks  us  how  our  suppliers  are  perfonning  we  are  now  a  little 
more  responsive”  (Honeywell  Interview).  Raytheon  gets  their  feedback  by  talking  to 
suppliers.  “We  try  to  benchmark  suppliers  against  their  competition  without  them 
knowing  who  their  competition  is”  (Raytheon  interview).  At  UTC,  the  suppliers  call 
purchasing  personnel  for  feedback  or  they  e-mail  them.  “If  they  get  good  ratings  we 
don’t  hear  from  them,  but  if  they  get  bad  ratings  they’ll  contact  us”  (UTC  Interview).  In 
United  Airlines’  case,  their  purchasing  personnel  are  reluctant  to  get  into  lengthy 
discussion  because  they  don’t  want  to  prolong  the  buying  process  by  having  to  go  into 
lengthy  detail  as  to  why  a  supplier  didn’t  win  (United  Airlines  Interview).  FedEx  doesn’t 
receive  much  feedback  from  suppliers  due  to  the  time  spent  up-front  in  explaining  to 
suppliers  how  the  scorecard  system  works  (FedEx  Interview).  Lockheed  Martin  has  an 
annual  conference  with  their  strategic  suppliers  where  concerns  are  addressed  (Lockheed 
Martin  Interview). 
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Question  7  “Would  you  categorize  most  of  your  contracts  as  fixed-price,  incentive  based, 
or  cost-plus?” 


Number 


Percent 


This  question  indicated  that  most  commercial  firms  use  fixed-price  contracts.  The 
government  offices  reported  cost-type  arrangements  as  the  most  common.  The  contract 
type  depends  greatly  on  what  type  of  buying  an  organization  is  engaged  in.  In  the 
government  system’s  acquisition  environment  the  supplier  is  subjected  to  more  risk.  This 
is  due  to  the  amount  of  research  and  development  associated  with  first  time  buys.  As  a 
result  the  government  employs  cost-type  arrangement  to  share  the  risk  with  the  supplier. 

It  is  hard  to  compare  these  commercial  sector  and  the  government  programs  with  this 
question.  An  interesting  note  is  that  Honeywell,  UTC,  and  FedEx  reported  that  they  were 
using  many  fixed-priced  regressive  contracts.  This  means  that  the  cost  is  reduced  each 
year  because  of  improvements  that  suppliers  are  expected  to  make  (UTC  Interview). 


Question  8  “Is  the  importance  of  price  tailored  for  each  acquisition?’ 


mix  with  performance/quality 
complexity/depends 


Number 


7 

4 

2 


Percent 


70.0  % 
40.0  % 
20.0  % 


n  =  10 


The  most  common  response  was  yes,  though  in  reality  firms  use  numerous 
approaches.  In  most  cases  there  was  an  increased  emphasis  on  non-price  related  factors. 
“In  the  case  of  buying  a  heads-up  display  or  electronics  you’re  buying  a  design,  and  price 
is  going  to  become  less  important”  (Delta  Airlines  Interview).  Raytheon  often  uses  a 
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30/30/30/10  scale.  They  put  30%  emphasis  on  quality,  30%  emphasis  on  on-time 
delivery,  30%  on  price,  and  a  10%  management  factor  (Raytheon  Interview).  Boeing 
states  that  price  is  most  important  during  OEM  buys,  but  if  they  are  developing  a  new 
system  then  they  examine  life-cycle  cost  which  includes  the  cost  of  ownership, 
maintenance  costs,  and  quality  concerns  (Boeing  Interview).  Much  of  the  weighting  for 
price  during  an  evaluation  depends  on  the  specifications  for  an  item.  It  is  often  not  the 
most  important,  but  if  the  requirements  are  well  defined,  it  should  be  (AFRL  Interview). 
Lockheed  Martin  stated  that  they  have  moved  entirely  to  a  best  value  approach,  even 
when  using  reverse  auctions  (Lockheed  Martin  Interview). 


Question  9  “Is  price  the  most  important  factor  in  evaluation?’ 


no/best  value 
depends 


Number 


7 

3 

2 


Percent 


70.0  % 
30.0  % 
20.0  % 


n  =  10 


Half  of  those  interviewed  answered  this  question  no.  However,  those  that  said  it 
was  a  best  value  approach  were  grouped  to  with  the  “no”  answers.  All  10  said  that  it  was 
a  factor  in  every  buy,  but  not  necessarily  the  most  important.  Honeywell  was  grouped  in 
the  yes  category  but  only  because  of  their  particular  scenario.  Often  times,  price  is  all 
that  remains  to  discriminate  based  on  the  high  standards  of  the  aerospace  industry 
(Honeywell  Interview).  Raytheon  explained  that  historically  price  has  been  pegged  as 
most  important;  however,  it  has  changed.  “We  really  try  an  emphasize  best  value,  if  I 
pay  10  cents  less  and  I  end  up  having  to  ship  it  back  it  becomes  more  expensive” 
(Raytheon  Interview).  A  UTC  representative  did  point  out  that  their  personal  opinion 
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was  that  price  wasn’t  most  important,  but  that  the  corporate  line  would  probably  be  “yes” 
(UTC  Interview). 


Post  Award  Evaluation 


Question  10  “How  is  post-award  evaluation  maintained  by  your  firm?’ 


rate/ratings/scorecard 

overall  program  management 
track  perfonnance 
time  &  delivery 


Number _ Percent 

5  50.0  % 

4  40.0  % 

4  40.0  % 

3  30.0  % 


n  =  10 


Many  of  the  companies  interviewed  have  instituted  scorecard  rating  systems  to 
evaluate  their  suppliers  after  contract  award.  Many  of  the  companies  are  still  in  the  early 
stages  of  developing  scorecards.  “We  have  a  scorecard  that  we  give  to  our  top  50  or  so 
suppliers  in  technical  operations  and  we  have  quarterly  meetings  to  discuss  the  numbers 
with  them  (Delta  Interview).  Much  of  this  question  seemed  to  be  dependent  on  whether 
this  was  the  first  time  a  company  was  dealing  with  a  supplier.  Honeywell  said  that  for 
new  products  they  look  at  management  capability,  project  plans,  schedule  targets,  process 
capability,  and  first-article  production  (Honeywell  Interview).  For  something  like 
microprocessors  from  Intel  we’re  not  going  to  have  much  leverage,  as  they’ll  tell  us  when 
they’re  coming  to  market  (Honeywell  Interview).  Raytheon  has  a  newly  established 
rating  system  which  establishes  thresholds  of  performance  (Raytheon  Interview).  If  a 
supplier  slips  below  the  threshold  they  are  put  on  probationary  status.  Raytheon  has  a 
newly  established  evaluation  tool  called  the  SRS  (Supplier  Rating  System).  Boeing  also 
has  a  thorough  scorecard  system  where  they  rate  suppliers  on  a  specific  procurement  and 
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roll  that  into  an  across  the  board  rating  (Boeing  Interview).  United  Airlines  expressed 
some  concern  that  they  are  not  doing  as  much  as  they  could  to  track  on-going  supplier 
performance  (United  Interview).  The  United  representative  reported  that  under  different 
financial  times  they  would  like  to  invest  in  a  supplier  rating  tool  but  that  currently  it  is 
not  possible.  UTC  has  a  monthly  scorecard  system  for  quality  and  delivery  information 
that  is  published  on  its  website  (UTC  Interview).  All  six  divisions  at  UTC  roll-up  their 
receipts  for  suppliers  and  they  consolidate  ratings  at  the  corporate  level.  FedEx  uses 
supply  chain  specialists  and  sourcing  managers  who  produce  ratings  on  a  monthly  basis 
via  a  scorecard  (FedEx  Interview).  Additionally,  there  are  supply  management 
specialists  who  maintain  a  portfolio  on  each  company  (FedEx  Interview).  Lockheed 
Martin  also  compiles  ratings  on  each  supplier  which  can  be  used  for  future  award 
determinations  (Lockheed  Martin  Interview).  They  have  a  STAR  supplier  program 
which  recognizes  excellent  performance  and  a  Supplier  Performance  Evaluation  & 
Action  Response  (SPEAR)  program  to  help  suppliers  recover  from  past  shortfalls  in  their 
ratings.  The  Government  program  offices  use  the  Contract  Performance  Assessment 
Reports  (CPARS)  rating  system  if  certain  threshold  levels  are  exceeded  (C-130  AMP 
Interview).  CPARs  are  not  required  for  smaller  purchases.  The  Government  does  have 
award-fee  determinations  which  are  kept  on  file;  however,  they  are  not  normally 
referenced  for  future  award  determinations  (C-130  AMP  Interview).  The  AFRL  does  not 
use  CPARs  all  too  often.  The  process  at  AFRL  is  very  informal  and  can  make  it  hard  to 
compare  suppliers  down  the  road  (AFRL  Interview). 
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Question  11a  “Is  the  process  automated  or  manual?’ 


manual  process 

delivery/quality  (automated  process) 
web  (automated  process) 


Number 


4 

3 

2 


Percent 


40.0  % 
30.0  % 
20.0  % 


n  =  10 


The  responses  for  this  question  were  somewhat  mixed.  Many  of  the  companies 
used  manual  retrieval  processes  while  others  had  databases  which  could  be  queried 
automatically  to  obtain  quality  and  delivery  information.  Delta  Airlines  has  a  semi- 
automated  system  which  extracts  data. 


Question  lib  “What  is  the  role  of  purchasing  personnel  in  this  process?’ 


database/spreadsheet  (purchasing  role) 
no  effort  (automatic) 


Number 

3 

2 


Percent 
30.0  % 
20.0  % 


n  =  10 


There  are  four  spreadsheets  for  on-time  delivery,  lead-time,  pricing,  and  quality 
(Delta  Interview).  The  buyer  then  consolidates  the  spreadsheets  into  one.  Honeywell 
uses  electronic  media  to  communicate  with  suppliers  regarding  their  quality,  parts 
rejected  over  parts  received,  and  on-time  delivery,  but  most  of  the  collection  and 
dissemination  is  manual  (Honeywell  Interview).  They  hope  to  have  a  web-based 
scorecard  by  next  year.  Raytheon’s  process  is  somewhat  automated  (Raytheon 
Interview).  There  is  an  incoming  inspection  loop  and  database  that  scores  the  supplier’s 
performance.  The  buyer  can  query  the  supplier  by  code  and  the  system  will 
automatically  download  performance  data  for  the  last  6  months,  1  year,  or  2  years 
(Raytheon  Interview).  At  Boeing  the  process  is  semi-automated.  The  database  requires 
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manual  input,  but  the  output  can  be  pulled  automatically  (Boeing  Interview).  The 
buyer’s  role  is  fairly  limited  (Boeing  Interview).  United’s  post-award  process  is  manual. 
If  they  don’t  hear  complaints  from  internal  users,  then  they’re  assumed  to  be  satisfied 
(United  Interview).  At  UTC  it  is  a  combination  of  both.  “It’s  automated  out  of  the 
business  units  but  it’s  a  manual  effort  to  consolidate  it  because  of  the  legacy  computer 
systems  we  still  use”  (UTC  Interview).  FedEx  also  uses  a  mix.  The  number  of  purchase 
orders  that  are  delivered  on-time  is  automatic,  but  other  infonnation  can  take  30  minutes 
to  an  hour  for  buyers  to  retrieve  (FedEx  Interview).  At  Lockheed  Martin  a  good  portion 
of  post-award  evaluation  is  automated.  The  monthly  quality  and  delivery  feedback  is 
automated  via  e-mail,  and  the  semi-annual  data  requires  the  buyers  to  fill  out  a  web  form 
(Lockheed  Martin).  Under  the  Government  programs  the  CPARs  and  award-fee 
determinations  are  the  responsibility  of  the  program  managers  (C-130  AMP  Interview). 
The  purchasing  personnel  provide  support  with  administrative  contract  information  only 
(AFRL  Interview). 

Question  12a  “How  much  time  (hours,  days)  is  required  to  complete  a  post-award 
evaluation?” 

_ Number _ Percent 

30  minutes  to  1  hour  4  40.0  % 

5  minutes  (how  much  time)  2  20.0  % 

1  month _ 1 _ 10.0  % 

n  =  10 

Responses  to  this  question  were  dependent  on  the  automation  level  of  a 
company’s  evaluation  system.  Delta  Airline’s  process  takes  about  a  half  hour  for  each  of 
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their  50  strategic  suppliers,  and  purchasing  personnel  can  update  infonnation  at  the  first 
of  the  month  or  at  least  quarterly  (Delta  Interview).  At  Boeing  the  process  takes  about  30 
minutes  each  month  to  compile  the  data  into  a  12  month  moving  average  (Boeing 
Interview).  At  FedEx  the  process  takes  only  a  couple  of  minutes  to  retrieve  the  data 
(FedEx  Interview).  The  monthly  reports  at  Lockheed  Martin  take  about  30  minutes  to 
complete  while  the  semi-annual  reports  are  more  time  consuming  (Lockheed  Martin 
Interview).  The  C-17  estimate  for  award-fee  reviews  was  in  excess  of  100  man-hours  (C- 
130  AMP).  However,  the  interviewee’s  C-17  experience  was  a  multi-billion  dollar  effort. 
Question  12b  “How  often  is  it  performed  (never,  monthly,  quarterly)?” 

_ Number _ Percent 

4  40.0  % 

3  30.0  % 

1  10.0  % 

n  =  10 


monthly 
quarterly 
not  performed 


The  most  common  response  was  monthly  or  quarterly.  Much  of  it  depended  on 
the  strategic  importance  of  a  particular  supplier.  The  AFRL  office  does  not  usually 
compile  this  information  for  science  and  technology  contracts.  In  other  government 
programs  it  is  compiled  during  an  award-fee  or  CPARs  write-up  on  a  quarterly  basis. 


Question  13  “Does  your  evaluation  after  award  affect  award-fees  or  bonuses  given  for 
above  average  performance?” 


not  given 
incentive/yes 
penalty _ 

n=  10 


Number _ Percent 

4  40.0  % 

4  40.0  % 

2  20.0  % 
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Four  of  the  eight  commercial  companies  answered  no  to  this  question.  Honeywell 
stated  that  its  suppliers  don’t  get  incentives  other  than  the  opportunity  to  get  more 
business  with  them  (Honeywell  Interview).  The  companies  that  said  “yes”  went  on  to 
say  that  very  few  of  their  contract  arrangements  were  tied  to  rating’s  incentives.  Delta 
said  that  they  sometimes  set  up  penalties  associated  with  performance  ratings,  but  this 
was  the  exception  rather  than  the  rule  (Delta  Interview).  The  government  program 
offices  do  use  incentive  type  contracts.  “There  are  very  few  incentives  in  our  contracts  as 
that  sort  of  thing  goes  in  cycles”  (AFRL  interview). 

Question  14  “What  factors  do  you  perceive  will  affect  the  success  or  failure  of  a 
contract?” 


_ Number _ Percent 

supplier  management/up-front  4  40.0  % 

specifications/communication  of  needs  3  30.0  % 

capability  2  20.0  % 

subcontractor  failure  2  20.0  % 


n  =  10 

This  question  indicated  that  supplier  management  was  extremely  important.  In 
addition  the  proper  specifications  were  also  a  pattern.  Up-front  involvement  with  the 
supply  chain  is  extremely  important  (Raytheon  Interview).  “Process  capability  is  critical 
but  we  are  also  extremely  concerned  with  how  a  supplier  manages  their  sub-tiers  because 
this  is  the  most  common  reason  for  supplier  delays  (Boeing  Interview).  Lockheed  Martin 
pushes  its  suppliers  to  manage  and  rate  their  suppliers  (Lockheed  Martin  Interview). 
Subcontracting  management  is  also  challenging  for  the  DoD,  because  the  Federal 
Acquisition  Regulation  (FAR)  gives  the  government  privity  with  its  prime  contractor  but 
not  its  subcontractor.  The  FAR  specifically  precludes  DoD  from  directing  the  actions  of 
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subcontractors.  There  is  a  CPARs  rating  for  subcontracting  management  which  can  be 
used  for  future  source  selections  to  help  gage  how  a  supplier  manages  its  subcontractors. 

General  Questions 

Question  15  “How  would  you  describe/define  strategic  sourcing? 


_ Number _ Percent 

supply  chain  6  60.0  % 

long-term  relationships  4  40.0  % 

team  effort  4  40.0  % 

partnering  3  30.0  % 

leveraging  spend _ 2 _ 20.0  % 


n=  10 

The  most  compelling  finding  from  this  question  was  the  unfamiliarity  of  this  term 
to  the  government  personnel.  This  term  is  often  preached  at  higher  headquarters  but  has 
not  yet  reached  the  program  offices  interviewed  (AFRL  Interview).  On  the  contrary,  the 
commercial  companies  have  even  gone  so  far  as  to  create  a  strategic  sourcing  section 
within  the  company.  “Strategic  sourcing  is  the  selection  of  long-term  suppliers  for 
sourcing  on  a  total  cost  basis.  It  is  aided  by  a  seven-step  process  on  the  web  that  each 
buyer  must  walk  through  in  order  to  source  an  item”  (Delta  Interview).  Honeywell 
defines  strategic  sourcing  as  leveraging  their  spend  to  get  the  best  prices  from  their 
suppliers  (Honeywell  Interview).  Raytheon  points  out  that  the  whole  element  of  strategy 
in  sourcing  is  to  find  differentiators  among  competitors.  Raytheon’s  other  key  points  are: 
1 )  The  folks  that  they  buy  the  most  from  are  brought  in  early  for  partnering,  and  2)  they 
ensure  that  they’ve  got  the  right  skill  sets  in-house  because  our  people  need  to  understand 
who  we  should  link  with  (Raytheon  Interview).  United  has  a  strategic  sourcing 
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department  and  their  representative  feels  that  it  shows  how  purchasing  has  changed.  “It 
is  essentially,  looking  at  more  than  price  and  establishing  long-term  relationships  that 
create  long-term  value  for  the  company”  (United  Interview).  UTC  uses  commodity 
management  teams  to  drive  the  supply  base  towards  commonality  among  the  divisions 
within  their  company  (UTC  interview).  Lockheed  Martin  defines  strategic  sourcing  as  a 
attempt  to  stratify  the  supply  base  at  the  top  level  of  management  (Lockheed  Martin 
Interview).  All  eight  commercial  companies  said  strategic  sourcing  applies  before  and 
after  award. 

Question  16  “What  3  things  could  you  have  done  better  in  pre-award  and  post-award 
evaluations  knowing  what  you  know  now?” 


_ Number _ Percent 

up-front  work  3  30.0  % 

objective  criteria/standard  process  2  20.0  % 

firm  specifications  2  20.0  % 

improved  ethics _ 2 _ 20.0  % 


n  =  10 

This  question  resulted  in  a  variety  of  answers.  The  four  “lessons  learned”  were: 

1)  up-front  work,  2)  objective  criteria  or  a  standard  process,  3)  firm  specifications,  and  4) 
ethics.  Delta  said  that  they  are  making  a  conscious  effort  to  make  decision  on  an 
objective  base.  “We  talk  about  what  we’re  trying  to  achieve  but  often  we  don’t  nail 
down  the  terms,  and  as  a  result,  we  often  select  a  supplier  based  on  what  feels  right” 
(Delta  Interview).  In  an  attempt  to  save  money,  Honeywell  explained  that  they  are  often 
guilty  of  misleading  a  supplier  by  letting  prospective  suppliers  conduct  self-assessments 
by  video  (Honeywell  Interview).  “When  a  new  supplier  gets  excited  about  doing 
business  with  us,  they  can  do  wonders  with  a  digital  video  camera  .  .  .  the  images  of  the 
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manufacturing  line  look  promising,  but  if  we  had  actually  visited  the  plant  we  would 
have  realized  they  didn’t  have  the  capability”  (Honeywell  Interview).  This  can  become 
an  ethical  challenge  for  a  “hungry”  supplier.  Raytheon  also  expressed  an  ethics  concern 
about  the  financial  viability  of  many  companies  (Raytheon  Interview).  “Ethics  is 
becoming  significant  in  pre-award  and  post-award  evaluations”  (Raytheon  Interview). 
Given  the  Enron  and  WorldCom  debacles,  there  is  reason  to  believe  this  trend  will 
continue.  Boeing  expressed  concern  that  their  organization  has  yet  to  institutionalize  the 
scorecard  system  (Boeing  Interview).  Their  push  is  to  get  “buy-in”  from  their  employees 
and  staff  to  utilize  their  supplier  evaluation  procedures.  United  Airlines  said  that  firm  or 
loose  specifications  are  often  the  most  common  reasons  a  contract  succeeds  or  fails 
(United  Interview).  The  Government  personnel  both  agreed  that  an  earlier  start  would 
enhance  a  source  selection.  “Getting  involved  with  the  Defense  Contract  Management 
Agency  (DCMA)  to  find  out  how  a  particular  division  of  a  major  corporation  is 
performing  would  take  some  of  the  guess  work  out  of  the  process”  (C-130  AMP 
Interview).  Another  interesting  comment  from  the  C-130  AMP  Interview  involved  the 
lack  of  experience  at  the  senior  management  level  regarding  past  performance 
evaluations.  “Program  offices  are  hesitant  to  assign  key  managers  to  participate  on  a 
source  selection  because  of  other  responsibilities,  and  as  a  result,  the  senior  leaders  do  not 
know  the  process  nor  can  they  provide  decision  making  assistance  to  the  source  selection 
evaluation  team  (SSET)  members”  (C-130  AMP  Interview). 
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Question  17a  “How  much  training  is  given  in-house  regarding  supplier  evaluations?” 

_ Number _ Percent 

7  80.0% 

3  30.0  % 


n=  10 


2-4  hours 
8  hours 


There  was  great  variability  in  this  answer.  A  few  of  the  firms  had  some  type  of 
training  to  discuss  supplier  evaluations;  however,  they  consolidated  it  with  the  annual 
training  for  purchasing  and  sourcing  personnel.  “A  year  or  two  back  we  wanted  all  the 
buyers  to  become  certified  purchasing  managers,  and  so  they  went  to  six  or  seven  days  of 
training  that  covered  a  bit  of  the  scorecard  process”  (Delta  Interview).  Honeywell 
requires  40  hours  of  training  on  their  business  processes  and  software  training  for 
OneSource,  which  is  used  by  purchasing  personnel  to  ensure  they  take  the  right  steps  in 
selecting  a  supplier  (Honeywell  Interview).  Raytheon  used  subject  matter  experts  to 
deploy  their  scorecard  system.  They  used  quality,  engineering,  and  supply  chain 
personnel  to  conduct  16  hours  of  training  (Raytheon  Interview).  Boeing  provides  three 
hours  of  training  for  pre-award  and  source  selections  and  another  four  hours  to  cover 
their  preferred  supplier  program  (Boeing  Interview).  UTC  provides  a  16  hour  class 
which  discusses  their  eight-step  sourcing  process  (UTC  Interview).  The  government 
personnel  receive  two  hours  of  training  on  EZ-Source,  which  is  a  database  tool  for 
compiling  past  performance,  cost  performance,  and  technical  information  for  source 
selections  (C-130  AMP  &  AFRL  Interview). 
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Question  17b  “What  type  of  training  is  given  in-house  regarding  supplier  evaluations?” 


_ Number _ Percent 

new  employee 
yearly  update 
software  training 

n  =  10 

There  was  not  much  variation  in  the  types  of  training  offered.  The  government 
and  commercial  companies  appear  to  employ  the  same  types  of  teaching  methods. 

Question  1 8  “Does  your  firm  use  multi-functional  teams  when  performing  pre-award  and 
post-award  evaluations  or  do  different  departments  (e.g.  engineering,  purchasing, 
marketing)  act  separately?” 

_ Number _ Percent 

yes,  ipt/multi-functional/commodity  10  100.0  % 

no _ 0 _ 0.0  % 

n=  10 

The  government  and  the  commercial  firms  all  used  the  integrated  product  team 
(IPT)  approach  when  conducting  evaluations.  Some  firms  did  not  call  it  an  IPT,  but  in 
principle  it  was  a  multi-disciplined  (i.e.  quality,  engineering,  supply,  finance, 
procurement)  approach.  Firms  are  not  likely  to  be  located  in  the  same  office,  but  they  do 
have  a  supply  chain  leader  and  a  business  leader  who  draws  in  resources  as  needed  (Delta 
Interview).  The  respondents  said  that  much  of  the  use  of  IPTs  would  be  determined  by 
the  complexity  of  the  buy,  or  the  frequency  of  the  buy.  If  it  was  a  repetitive  buy,  then  an 
IPT  might  not  be  required.  “The  best  situation  is  one  where  you  sit  people  together:  The 
decision  making  is  real  time”  (Raytheon  Interview).  The  government  personnel  felt  that 
the  IPT  approach  works  really  well.  “Years  ago  there  were  concerns  that  if  an  individual 


4  40.0  % 

3  30.0  % 

2  20.0  % 
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was  matrixed  out  to  a  specific  program  that  they  might  lose  support  from  the  home  office, 

but  I  don’t  think  that’s  been  the  case”  (C-130  AMP  Interview). 

Question  19  “How  many  individuals  typically  participate  in  a  pre-award  or  post-award 
evaluation  (e.g.  $50M  acquisition)?” 

_ Number _ Percent 

core  4-10  people  4  40.0% 

10-20  3  30.0  % 

unknown _ 3 _ 30.0  % 

n  =  10 

$50M  was  used  to  simplify  this  question.  The  StatPac  program  did  not  generate 
numbers  in  its  analysis  and  as  such  some  examples  are  provided.  “20  or  30  people 
wouldn’t  surprise  me,  but  I  would  definitely  say  that  we  use  a  core  of  6  to  10  people” 
(Delta  Interview).  “We  would  use  a  core  of  four  people:  a  commodity  lead,  a  quality 
representative,  and  an  engineering  representative;  however,  it  would  definitely  touch  a  lot 
more”  (Honeywell).  “I  would  say  a  core  of  4  to  10  people  would  be  used”  (UTC 
Interview).  The  government  teams  were  both  estimated  at  around  15  to  20  people  (C-130 
AMP  and  AFRL  Interview).  These  answers  did  not  produce  any  significant 
inconsistency  between  the  government  and  commercial  companies. 

Question  20  “Would  you  characterize  your  relationships  with  suppliers  as  short-tenn  or 
long-term? 

_ Number _ Percent 

9  90.0  % 

1  10.0  % 

0 _ 0.0  % 


n  =  10 


long-term 

mix 

short 
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The  majority  of  those  questioned  said  that  their  relationships  were  becoming  more 
and  more  long-term.  The  mixed  response  came  from  AFRL.  “If  we  are  doing  research  in 
the  propulsion  division  it’s  long  term,  but  if  its  Small  Business  Innovative  Research 
(SBIR)  contractors  it’s  usually  short-term”  (AFRL  Interview).  This  is  because  of  the 
turnover  in  the  SBIR  program. 

Question  2 1  “Has  your  relationship  with  your  suppliers  improved  or  suffered  because  of 
your  evaluation  procedures?” 

_ Number _ Percent 

improved/expect/relationship  8  80.0  % 

unchanged/ suffered _ 2 _ 20.0  % 

n  =  10 

8  of  the  10  cases  said  that  their  relationships  had  improved  with  their  suppliers. 
Delta  and  Honeywell  both  felt  that  their  relationships  were  unchanged  or  had  suffered. 
“Delta  has  brought  some  supplies  back  in-house  and  now  we  are  competitors  with  some 
of  our  suppliers”  (Delta  Interview).  Honeywell  says  that  they  are  still  demanding 
towards  suppliers  and  that  some  may  even  be  agitated  by  the  fact  that  Honeywell  expects 
them  to  now  share  more  of  the  costs  (e.g.  inventory)  (Honeywell  Interview).  UTC 
expects  that  their  implementation  of  OpenRatings  will  be  pivotal  in  detennining  whether 
suppliers  approve  of  their  evaluation  procedures.  “The  suppliers  like  our  scorecards 
because  it  provides  them  feedback  on  a  more  consistent  basis”  (Lockheed  Martin 
Interview).  “If  a  supplier  doesn’t  receive  our  e-mail  scorecard  reports  then  we  definitely 
hear  from  them”  (FedEx  Interview).  The  government  personnel  have  seen  improvement 
as  far  as  how  suppliers  perceive  the  source  selection  process.  “Our  past  performance 
evaluations  are  now  more  open”  (AFRL  Interview).  This  is  due  in  large  part  to  the  de- 
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briefings  that  are  given  to  the  successful  and  unsuccessful  offerors  for  a  given  source 
selection  (C-130  AMP). 


“Commercial  Company”  Only  Questions 

Question  1C  “What  is  your  impression  of  the  current  DoD  past  perfonnance  and  risk 
assessment?” 

_ Number _ Percent 

4  50.0  % 

3  37.5  % 

2  25.0  % 


n  =  8 


thorough  process 
fair 

time  consuming 


The  responses  were  not  mutually  exclusive.  Some  companies  had  an  opinion  and 
others  did  not.  I  was  surprised  to  discover  that  many  of  the  commercial  companies 
thought  that  the  DoD  source  selection  process  was  effective.  Most  admitted  that  they 
could  not  spend  that  amount  of  time  or  money  for  such  an  endeavor,  but  said  that  it  was 
necessary  to  satisfy  the  taxpayers.  “My  impression  is  it  takes  a  lot  of  time  upfront,  but 
that  it’s  a  very  thorough  and  necessary  effort”  (Delta  Interview).  “I  think  it’s  fairly 
consistent  with  what  we  do,  and  we  have  even  tried  to  mirror  our  process  to  match  the 
DoD’s  in  some  ways”  (Raytheon  Interview).  One  individual  commented  that  some 
commercial  companies  do  not  want  to  do  business  with  the  DoD  because  of  the 
administrative  burden  it  places  on  a  potential  supplier  (United  Airlines  Interview). 


49 


Question  2C  “What  is  your  impression  of  third-party  rating  systems  (e.g.  OpenRatings)?” 


_ Number _ Percent 

only  heard  of  it/no  impression  5  62.5  % 

just  another  evaluation  tool  3  40.0  % 

wouldn’t  put  100%  confidence  in  it  2  25.0  % 

useful  for  financial  measures  2  25.0  % 


n  =  8 

More  than  half  of  those  interviewed  had  heard  of  the  OpenRatings  system.  UTC 
was  the  only  company  that  had  purchased  the  OpenRatings  scorecard  option  but  others 
such  as  Honeywell  were  using  the  OpenRatings  as  a  tool  in  their  evaluations.  “We  use 
them  as  an  alert  mechanism  because  some  of  commodity  managers  don’t  have  the  time  to 
look  at  financials”  (Honeywell  Interview).  “It  lets  the  buyers  set  up  a  watch  list  for  those 
buyers  that  are  managing  many  suppliers  and  our  impression  of  it  has  very  positive” 
(Honeywell  Interview).  “We  don’t  put  100%  faith  in  OpenRatings  but  we  use  it  to  drive 
some  internal  activities”  (UTC  Interview).  Other  companies  are  a  little  more  hesitant  to 
use  such  a  system.  “The  system  at  Lockheed  Martin  is  already  so  automated  that 
OpenRatings  would  not  provide  much  benefit.  I  see  its  biggest  payoff  to  the  smaller 
companies  that  don’t  have  the  computer  programming  resources”  (Lockheed  Martin 
Interview).  “I  hear  it’s  pretty  good.  It  looks  attractive  but  as  upgrades  become  necessary 
it  could  become  expensive.  There’s  also  so  much  to  control,  and  even  though 
OpenRatings  would  be  a  partner  they’re  still  in  it  to  make  money”  (Raytheon  Interview). 
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Question  3C  “If  your  process  is  automated,  who  designed  the  software?” 

_ Number _ Percent 

4  50.0% 

2  25.0  % 

2  25.0  % 


in-house  design 
help  from  a  third  party 
COTS  software _ 


n  =  8 

Half  of  the  companies  created  the  system  in-house.  None  of  the  companies 
interviewed  had  turned  over  responsibility  for  ratings  to  a  third-party  company.  UTC  and 
Honeywell  use  OpenRatings,  but  only  in  addition  to  their  in-house  system  (Honeywell 
and  UTC  Interviews).  Raytheon  designed  their  automated  rating  system  in-house  with 
the  help  of  Price-Waterhouse-Cooper  (Raytheon  Interview).  Others  such  as  FedEx  and 
Lockheed  use  COTS  software  or  a  homemade  windows-based  program. 


Question  4C  “Do  you  maintain  one  rating  for  each  supplier  which  covers  all  contracts  or 
are  they  given  a  rating  which  is  based  on  their  performance  on  one  particular  contract?” 

_ Number _ Percent 

all  contracts  (aggregate  roll-up)  6  75.0% 

particular  contract  5  62.5  % 

by  division _ 2 _ 25.0  % 

n  =  8 


This  question  is  extremely  important  in  determining  the  capability  and  level  of 
effort  required  by  a  supplier  rating  system.  Many  companies  are  still  trying  to  detennine 
if  it  is  necessary  to  give  ratings  for  each  contract  or  if  an  across  the  board  approach  is 
sufficient.  Honeywell,  Delta,  and  Boeing  use  across  the  board  roll-ups  for  divisions. 
Boeing  is  currently  investigating  ways  to  break  their  rating  down  into  individual  contracts 
(Boeing  Interview).  Raytheon,  FedEx,  UTC,  and  Lockheed  Martin  can  retrieve 
information  on  particular  contracts  but  often  roll-up  their  infonnation  to  division  level. 


51 


Question  5C  “Does  your  company  do  business  in  the  aircraft  or  manufacturing 
industries?” 


_ Number _ Percent 

yes 
no 


8  100.0  % 

0  0.0  % 


n  =  8 


This  question  was  originally  designed  to  capture  companies  that  might  not  have 
been  aircraft  or  manufacturing  industries.  In  all  8  cases  the  firms  were  government 
aircraft  contractors,  airline  companies,  or  air  shipment  specialists. 


Question  6C  “What  regulations  does  your  company  have  which  outline  the  pre-award 
evaluation  process?” 


procedures/ guidelines/ guide 
web-based 

policies/standard _ 

n  =  8 


Number _ Percent 

5  62.5  % 

3  37.5  % 

2  25.0  % 


Procedures  and  policies  are  not  lengthy  documents  in  comparison  with  the 
Federal  Acquisition  Regulation.  “Because  it’s  web-based  there  isn’t  a  ton  of  procedures 
around  it  as  it’s  an  internal  document  that  is  published  or  sent  via  e-mail  that  is 
approximately  10  pages  long”  (Raytheon  Interview).  Boeing  has  PROs  (Enterprise 
Procedures)  that  address  supplier  performance  measurement,  supplier  certification,  and 
strategic  sourcing  (Boeing  Interview).  In  most  cases  Boeing  will  keep  PROs  under  30 
pages.  United  Airlines’  entire  purchasing  regulation  is  web-based  and  only  15  pages 
long. 
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Government  Only 

Question  1G  “What  is  your  perception  of  the  FAR's  restrictions  on  the  pre-award 
evaluation  process?” 


not  restricted  by  FAR 


Number  Percent 


2  100.0  % 


n  =  2 


The  government  personnel  that  were  interviewed  did  not  find  the  FAR  restrictive 
when  applied  to  source  selection  evaluations.  “Most  of  the  guidance  that  we  used  in 
carrying  out  our  PRAG  was  driven  down  from  AFMC  Headquarters  or  SAF/AQC”  (C- 
130  AMP  Interview).  “Most  of  the  procedures  aren’t  in  the  FAR  but  come  down  from 
headquarters”  (AFRL  Interview). 

Question  2G  “How  many  protests  have  you  received  on  source  selections?” 

_ Number _ Percent 

none _ 2 _ 100.0  % 

n  =  2 

Neither  of  the  two  source  selection  cases  received  a  protest.  There  was  a  strong 
belief  that  the  C-130  AMP  program  did  not  receive  a  protest  because  of  the  upcoming 
Joint  Strike  Fighter  award  (C-130  AMP  Interview). 

Question  3G  “If  this  process  could  be  automated  would  it  alter  your  perception  of  the 
DoD  evaluation  process?” 

_ Number _ Percent 

yes/communication/subjective _ 2 _ 100.0  % 

n  =  10 
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The  Government  personnel  felt  that  the  scoring  could  be  automated;  however,  it 
might  kill  the  fundamental  benefit  of  communication  between  the  teams  (C-130  AMP). 
The  government  individual  interviewed  reflected  on  the  comments  from  General  Kadish, 
a  former  program  director  for  the  C-17,  who  said  that  if  ratings  are  quantified  then 
numbers  are  all  that  will  improve  (C-130  AMP  Interview).  The  thought  process  is  that 
communication  between  the  government  and  its  contractor/supplier  is  the  most  critical 
aspect  of  a  program’s  success  or  failure  (C-130  AMP  Interview).  The  AFRL 
representative  stated  that  there  is  a  lot  of  information  that  can  not  be  quantified  in  an 
automated  rating  and  went  on  to  say  that  subjective  ratings  can  be  a  good  thing  (AFRL 
Interview). 

Question  4G  “Does  the  program  office  currently  maintain  a  performance  file  for  each  of 
its  contractors?” 

_ Number _ Percent 

no _ 2  100.0  % 

n  =  2 

The  government  can  retrieve  perfonnance  information  on  a  particular  supplier  if  it 
is  currently  under  contract  in  an  award-fee  arrangement  (C-130  AMP  Interview).  “We 
also  have  CPARs  for  some  larger  contracts”  (C-130  AMP  Interview).  “Currently,  we  are 
only  capturing  performance  data  for  individual  contracts.  What  I’d  really  like  to  see  is  a 
roll-up  so  that  I  could  assess  Boeing’s  St.  Louis  division.  Currently  there  is  no  means  for 
the  program  offices  to  roll-up  their  data”  (C-130  AMP  Interview). 


54 


V.  Discussion  and  Recommendations 


The  purpose  of  this  research  was  to  examine  the  feasibility  of  USAF  acquisition 
personnel  using  automated  supplier  evaluation  systems  in  lieu  of  past  perfonnance 
evaluations  and  award-fee  boards.  The  source  selection  process  and  award-fee 
determinations  require  valuable  time  for  the  acquisition  professional.  It  is  not  uncommon 
for  the  source  selection  team  to  spend  six  months  evaluating  past  performance  only  to 
find  that  all  suppliers  are  equal  in  terms  of  past  performance.  If  the  goal  is  to  find 
discriminators  between  prospective  contractors,  then  there  may  be  a  cheaper  way.  The 
focus  of  this  effort  was  to  examine  potential  categories  (quantitative  and  qualitative  data) 
that  could  be  assessed  for  contract  award  or  for  evaluation  after  award. 

At  the  outset,  the  intent  was  to  determine  if  the  Purchasing  and  Supply  Chain 
Management  (PSCM)  program  could  utilize  supplier  performance  ratings  in  hopes  of 
identifying  capable  suppliers  at  a  much  faster  pace.  Industry  seems  to  have  a  positive 
attitude  toward  DoD’s  current  evaluation  system.  Most  companies  thought  that  although 
the  process  was  time  consuming,  it  was  effective.  Most  companies  interviewed  appeared 
to  agree  with  the  DoD  evaluation  system  because  of  public  law  and  taxpayer  scrutiny.  In 
many  ways  they  try  to  follow  what  DoD  does  in  these  areas.  However,  because  these 
firms  are  motivated  by  profit,  they  must  find  ways  to  expedite  and  cut  costs  during 
evaluations.  The  question  is,  how  far  is  the  USAF  acquisition  community  or  Congress 
willing  to  go  in  order  to  cut  costs?  Is  the  FAR  going  to  incorporate  an  automated  rating 
system  or  can  the  USAF  institute  such  a  system  on  its  own? 
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This  chapter  addresses  this  research  effort’s  conclusions,  benefits,  limitations,  and 
suggestions  for  future  research.  All  of  the  conclusions  will  be  presented  as  the  interview 
questions  were  mapped  to  the  research  questions  from  chapter  3. 

RECOMMENDATIONS/CONCLUSIONS 

Primary  Research  Question  1  asked  “can  USAF  acquisition  personnel  use  supplier 
ratings  in  lieu  of  conducting  award-fee  boards  or  past  performance  evaluations?”  This 
question  was  answered  through  a  series  of  investigative  questions. 

Interview  Question  (IQ)  10  asked  “How  is  post-award  evaluation  maintained  by 
your  firm/program  office?”  Many  of  the  companies  interviewed  reported  having 
instituted  scorecard  rating  systems  to  evaluate  their  suppliers  after  contract  award  and 
others  are  in  the  early  stages  of  developing  scorecards.  In  contrast,  the  government  uses 
previous  source  selections  and  headquarters  command  or  SAF/AQC  guides  to  model 
their  evaluations. 

IQ  1 1  asked  “is  the  process  automated  or  manual?”  What  is  the  role  of 
purchasing  personnel  in  this  process?  The  responses  for  this  question  were  somewhat 
mixed.  Many  companies  used  manual  retrieval  processes  while  others  had  databases 
which  could  be  queried  automatically  to  obtain  quality  and  delivery  information.  The 
role  of  the  purchasing  personnel  is  limited  to  data  retrieval. 

IQ  4C  (C=Contractor  Only)  asked  “do  you  maintain  one  rating  for  each  supplier 
which  covers  all  contracts  or  are  they  given  a  rating  which  is  based  on  their  performance 
on  one  particular  contract?”  Many  companies  are  still  trying  to  determine  if  it  is 
necessary  to  give  ratings  for  each  contract  or  if  an  across  the  board  approach  is  sufficient. 
Honeywell,  Delta,  and  Boeing  use  across  the  board  roll-ups  for  divisions.  Raytheon, 
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FedEx,  UTC,  and  Lockheed  Martin  can  retrieve  information  on  particular  contracts  but 
often  roll-up  their  information  to  division  level. 

IQ  16  asked  “what  3  things  could  you  have  done  better  in  pre-award  and  post¬ 
award  evaluations  knowing  what  you  know  now?”  This  question  resulted  in  a  variety  of 
answers.  The  four  most  prevalent  were:  1)  up-front  work,  2)  objective  criteria  or  a 
standard  process,  3)  firm  specifications,  and  4)  eliminating  ethical  concerns. 

I  would  recommend  that  the  USAF  acquisition  community  devise  an  automated 
scorecard  system  to  track  suppliers  on  contracts  above  $25K.  If  such  a  system  were 
intended  for  small  purchases  it  could  become  extremely  time  consuming  on  small 
repetitive  buys.  Although  my  research  was  limited  to  two  program  offices  at  the  product 
center  level,  an  automated  system  could  also  provide  infonnation  for  depot  and 
operational  buying  units.  Most  commercial  companies  are  moving  in  the  automated 
scorecard  direction  and  those  that  have  implemented  scorecard  ratings  perceive  them  to 
be  extremely  effective.  Potential  suppliers  as  well  as  current  suppliers  are  welcoming  the 
feedback  they  receive  from  these  companies  as  a  way  to  see  how  well  they’re  doing. 

The  implications  for  the  USAF  are  that  it  would  have  to  be  incorporated  with  a 
standard  procurement  system.  The  system  would  need  visibility  of  all  individual  contract 
ratings  for  a  particular  supplier  and  an  aggregate  rating  for  pre-award  assessments.  If 
there  are  post-award  implications,  then  only  the  ratings  for  a  specific  contract  would  be 
queried.  The  caveat  with  any  government  system  would  be  to  maintain  subjective 
ratings.  Quantitative  ratings  are  effective  but  often  result  in  a  different  outcome  than 
desired.  This  is  why  General  Kadish  (C-17  SPO  Director)  was  hesitant  to  make  all 
ratings  quantitative.  His  thought  process  was  that  only  the  quantifiable  numbers  on  paper 
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would  improve.  When  ratings  become  entirely  quantitative  the  qualitative  factors  such  as 
program  management  and  team  communication  will  not  improve  (C-130  AMP). 

Research  supports  General  Radish’s  fear:  “Whether  dealing  with  monkeys,  rats,  or 
human  beings,  it  is  hardly  controversial  to  state  that  most  organisms  seek  information 
concerning  what  activities  are  rewarded,  and  then  seek  to  do  (or  at  least  pretend  to  do) 
those  things,  often  to  the  virtual  exclusion  of  activities  not  rewarded”  (Kerr,  1975:769). 

Subsidiary  Question  2\  What  are  the  similarities  between  USAF  rating  categories  and 
commercial  rating  categories? 

IQ  1  asked  “what  are  the  categories/factors  of  assessment  used  for  evaluating  past 
performance?  How  were  these  categories  constructed  (i.e.  industry  standard,  from 
within)?  The  most  common  factor  was  on-time  delivery.  Quality  and  past  performance 
were  also  important.  In  this  case  some  companies  mentioned  price  or  cost,  and  although 
they  are  not  truly  the  same,  the  terms  were  joined  for  simplification.  How  the  categories 
were  constructed  was  not  significant. 

IQ  2  asked  “is  your  rating  system  quantitative  (e.g.  late  delivery  in  terms  of  #  of 
days)  or  qualitative  (e.g.  quality  performance)?  This  question  produced  an  interesting 
difference  between  commercial  companies  and  government  organizations.  The 
commercial  companies  expressed  a  desire  to  make  their  rating  categories  as  quantitative 
as  possible.  However,  the  commercial  companies  explained  that  a  mixture  of  the  two 
was  more  common.  On  the  contrary,  the  government  organizations  explained  the 
importance  of  qualitative  (subjective)  ratings. 

IQ  3  asked  “how  many  days  on  average  does  it  take  you  to  award  a  contract  (e.g. 
clock  starts  when  a  requirement  or  funding  document  is  received)?”  All  of  those 
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interviewed  asked  for  a  dollar  value  in  order  to  answer  the  question.  I  told  them  to  tell 
me  how  long  it  would  take  them  to  complete  a  $50M  buy  from  cradle-to-grave.  I  got 
answers  ranging  from  3  weeks  to  one  year.  Overall,  the  commercial  companies  were 
much  faster. 

IQ  4  asked  “has  your  pre-award  evaluation  process  changed  over  the  last  5 
years?”  Most  companies  stated  the  process  had  become  more  automated.  Delta  Airlines 
stated  that  they  process  has  changed  in  the  way  they  collect  and  distribute  information  to 
a  supplier  (Delta  Airlines).  I  noticed  that  companies  such  as  United,  Honeywell,  and 
FedEx  are  doing  reverse  auctions  with  p re-qualified  vendors.  Raytheon  stated  that  they 
have  changed  from  a  tool  and  process  standpoint... we’ve  gotten  more  electronic  and 
web-enabled”  (Raytheon).  The  government  personnel  stated  that  changes  over  the  last 
five  years  have  been  minimal  (AFRL). 

IQ  7  asked  “would  you  categorize  most  of  your  contracts  fixed-price,  incentive 
based,  or  cost-plus?”  This  question  indicated  that  most  commercial  firms  use  fixed-price 
contracts.  The  government  offices  reported  cost-type  arrangements  as  the  most  common. 
Most  of  it  depends  on  what  type  of  buying  an  organization  is  engaged  in. 

IQ  8  asked  “is  the  importance  of  price  tailored  for  each  acquisition?”  The  most 
common  response  was  yes,  but  it  depends.  In  most  cases  there  was  an  increased 
emphasis  on  non-price  related  factors. 

IQ  9  asked  “is  price  the  most  important  factor  in  evaluation?”  Half  of  those 
interviewed  answered  this  question  no.  However,  those  that  said  it  was  a  best  value 
approach  were  grouped  with  the  “no”  answers.  All  10  said  that  it  was  a  factor  in  every 
buy,  but  not  necessarily  the  most  important. 
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IQ  17  asked  “how  much  training  is  given  in-house  regarding  supplier 
evaluations?”  A  few  of  the  firms  had  some  type  of  training  to  discuss  supplier 
evaluations;  however,  they  consolidated  it  with  the  annual  training  for  purchasing  and 
sourcing  personnel. 

IQ  1 8  asked  “does  your  firm  use  multi-functional  teams  when  performing  pre¬ 
award  and  post-award  evaluations  or  do  different  departments  (e.g.  engineering, 
purchasing,  marketing)  act  separately?”  The  government  and  the  commercial  firms  all 
used  the  integrated  product  team  (IPT)  approach  when  conducting  evaluations.  Some 
firms  did  not  call  it  an  IPT,  but  in  principle  it  was  a  multi-disciplined  (i.e.  quality, 
engineering,  supply,  finance,  procurement)  approach. 

The  USAF  rating  categories  are  in  line  with  that  of  commercial  companies.  There 
is  little  evidence  to  indicate  that  there  is  a  distinct  difference  between  the  commercial 
company  and  USAF  evaluation  categories.  The  only  difference  is  that  the  USAF  system 
accounts  for  additional  qualitative  ratings.  If  evaluations  became  entirely  quantitative 
then  each  source  selection  might  not  be  a  true  “best-value”  (AFRL). 

On-time  delivery  and  quality  were  the  most  common  factors  examined  during  an 
evaluation.  Where  the  government  falls  short  is  in  its  ability  to  automatically  track  the 
delivery  or  receipt  of  an  item.  At  some  of  the  commercial  companies  the  receiving  dock 
automatically  inputs  delivery  information  into  a  database  to  be  pulled  when  assessing  a 
supplier’s  actual  delivery  performance.  The  same  can  be  said  about  quality  at  inspection. 
The  receiving/inspection  personnel  will  input  quality  defects  or  acceptance  rates.  In  the 
government  environment  purchasing  personnel  unaware  of  delivery  until  they  receive  an 
invoice  or  the  user  calls  to  complain  about  the  quality  of  a  commodity  or  service. 
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In  terms  of  award-fee  boards  it  was  extremely  hard  to  compare  the  categories 
between  commercial  companies  and  the  USAF  program  offices.  Commercial  companies 
predominately  use  fixed-priced  arrangements.  Very  few  of  them  have  incentives  to 
exceed  the  minimum  requirement  of  a  contract  other  than  more  business.  The  one 
recommendation  for  improvement  in  the  post-award  government  categories  would  be  to 
become  more  quantitative  in  the  three  areas  of  cost  performance,  time  of  delivery,  and 
schedule  (AFMC  Award-Fee  Guide,  2000:83). 

Subsidiary  Question  3:  Can  cost-avoidance  savings  be  quantified  if  either  were 
replaced? 

IQ  3  asked  “how  much  time  (hours,  days)  is  required  to  complete  a  pre-award 
evaluation?”  For  a  $50M  project  I  got  answers  ranging  from  3  weeks  to  one  year. 

Overall,  this  answer  was  dependent  upon  the  urgency  of  the  requirement. 

IQ  5  asked  “have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your 
evaluation  process?”  Many  of  the  answers  for  this  question  were  not  in  put  terms  of 
dollars.  Only  UTC  reported  a  $1B  savings  through  2001  on  the  product  side  to  Wall 
Street  and  a  $700  level  savings  by  the  year  2004  (UTC). 

IQ  12a  asked  “how  much  time  (hours,  days)  is  required  to  complete  a  post-award 
evaluation?”  This  question  was  extremely  dependent  on  how  automated  a  company’s 
evaluation  system  was.  Delta  Airline’s  process  took  about  a  half  hour  for  each  of  their  50 
strategic  suppliers  (Delta). 

IQ  12b  asked  “how  often  are  post  award  evaluation  performed  (never,  monthly, 
quarterly)?”  Some  of  the  companies  can  update  this  information  at  the  first  of  the  month 
but  the  typical  requirement  was  to  update  rating  information  quarterly.  At  Boeing  the 
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process  takes  about  30  minutes  each  month  to  compile  the  data  into  a  12  month  moving 
average  (Boeing). 

IQ  19  asked  “how  many  individuals  typically  participate  in  a  pre-award  or  post¬ 
award  evaluation  for  $50M?”  “20  or  30  people  wouldn’t  surprise  me,  but  I  would 
definitely  say  that  we  use  a  core  of  6  to  10  people”  (Delta).  “We  would  use  a  core  of  four 
people:  a  commodity  lead,  a  quality  representative,  and  an  engineering  representative; 
however,  it  would  definitely  touch  a  lot  more”  (Honeywell).  The  government  teams  were 
both  estimated  at  around  15  to  20  people  (C-130  AMP  and  AFRL).  These  answers  did 
not  produce  any  significant  inconsistency  between  the  government  and  commercial 
companies. 

Government  cost-avoidance  might  be  quantified  if  both  source  selections  and 
award-fee  evaluations  were  less  time  consuming.  When  source  selection  questionnaires 
concerning  supplier  past  performance  are  sent  to  government  contracting  officers  and 
program  managers  they  are  returned  with  quality,  on-time  delivery,  and  management 
information.  This  effort  is  time  consuming  for  a  PRAG  or  source  selection  team.  This 
was  evidenced  by  the  $15M  AFRL  EBS  acquisition  which  took  1  year  to  complete 
(AFRL).  If  quantitative  and  qualitative  information  could  be  pulled  from  a  supplier 
scorecard  database  then  questionnaires  would  become  unnecessary.  Additionally,  in  the 
case  of  the  C-17  award-fee  board  over  100  man  hours  was  spent  each  quarter  in  assessing 
McDonnell  Douglas’s  (now  Boeing)  performance  (C-130  AMP).  Some  of  this  time 
would  be  saved  if  ratings  were  automated. 

Subsidiary  Question  4\  What  acquisition  refonn  initiatives  or  regulations  would  aid  or 
hind  USAF  application  of  a  revised  perfonnance  evaluation  system? 
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IQ  1G  (G=Govemment  Only)  asked  “what  is  your  perception  of  the  FAR’s 
restrictions  on  the  pre-award  evaluation  process?”  The  government  personnel  that  were 
interviewed  did  not  find  the  FAR  restrictive  when  applied  to  source  selection  evaluations. 
“Most  of  the  guidance  that  we  used  in  carrying  out  our  PRAG  was  driven  down  from  Air 
Force  Materiel  Command  Headquarters  or  SAF/AQC”  (C-130  AMP).  “Most  of  the 
procedures  aren’t  in  the  FAR  but  come  down  from  headquarters”  (AFRL  Interview). 

IQ  6C.  (C=Contractor  Only)  asked  “what  regulations  does  your  company  have 
which  outline  the  pre-award  evaluation  process?”  Procedures  and  policies  are  not 
lengthy  documents  in  comparison  with  the  Federal  Acquisition  Regulation.  “Because  it’s 
web-based  there  isn’t  a  ton  of  procedures  around  it.  It’s  an  internal  document  that  is 
published  or  sent  via  e-mail.  Buyers  do  have  a  desktop  guide  that  is  10  pages  long” 
(Raytheon). 

My  research  fails  to  provide  evidence  that  refonn  initiatives  or  FAR  regulations 
would  impede  a  more  automated  performance  evaluation  system.  In  fact,  the  FAR  is  not 
as  restrictive  as  was  hypothesized.  Most  of  the  guidance  for  evaluations  comes  from  Air 
Force  Materiel  Command  Contracting  Division  (AFMC/PK)  or  the  Deputy  Assistant 
Secretary  (Contracting)  of  the  Air  Force  (SAF/AQC).  Much  of  the  guidance  is  “lessons 
learned”  from  previous  source  selections  (AFRL  Interview). 
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Benefits  and  Contributions  of  Research 


This  research  effort  resulted  in  evidence  to  support  the  continuing  push  for  improving 
the  way  we  evaluate  suppliers.  These  findings  will  be  a  source  of  infonnation  for  AF/IL 
and  headquarter  personnel  in  their  efforts  to  devise  a  scorecard  system  for  the  USAF.  An 
external  benefit  of  this  research  will  also  be  to  the  companies  that  participated.  They  will 
be  able  to  gauge  their  efforts  in  automating  their  performance  evaluation  systems  with 
that  of  their  competitors.  The  companies  and  program  offices  interviewed  were 
extremely  candid  in  their  conversations  with  me  and  did  not  withhold  infonnation  based 
on  a  concern  that  others  might  try  to  imitate  them.  Having  said  that,  the  benefits  of 
automated  evaluation  systems  seems  positive  in  the  eyes  of  purchasing/sourcing 
personnel. 

The  results  of  this  study  provide  the  USAF  with  more  infonnation  which  allows  us  to 
operate  more  and  more  like  that  of  a  profit-driven  company.  The  bottom  line  is  that  we 
can  not  continue  to  spend  a  months  and  sometimes  years  to  evaluate  a  potential  supplier, 
we  need  a  rapid  capability  that  is  more  cost  effective. 

Limitations 

The  main  limitation  of  this  methodology  deals  with  sample  bias.  The  commercial 
company  sample  was  chosen  from  the  2002  Fortune  500  “top  5”  of  the  government  and 
airline/transportation  industries.  The  bias  results  in  the  aerospace  emphasis  that  was 
placed  on  the  study.  Many  of  the  USAF  acquisitions  are  not  aircraft  related. 

Additionally,  all  of  the  firms  that  were  interviewed  were  large  businesses.  The  USAF 
does  a  significant  amount  of  buying  from  small  businesses  without  the  IT  infrastructure 
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that  these  companies  have.  The  government  bias  stems  from  the  sample  being  taken  at 
Aeronautical  Systems  Center  (ASC)  at  Wright-Patterson  AFB.  Each  product  center, 
depot  buying  activity,  or  operational  unit  has  varying  requirements.  Aircraft  program 
offices  are  buying  aircraft,  aircraft  components,  or  modifications.  Depots  support  the 
overhaul  and  spare  parts  operations,  while  operational  units  support  the  efforts  for  a 
particular  base.  This  makes  my  study  limited.  Results  of  a  study  taken  from  a  sampling 
frame  can  only  be  generalized  to  that  sampling  frame  (Dooley,  2001 : 127).  This  makes  it 
difficult  to  make  generalizations  directed  towards  an  operational  buyer  who  specializes  in 
base  services  (e.g.  cafeteria,  lawns)  when  the  research  was  focused  on 
airline/transportation  companies,  government  contractors,  and  government  program 
offices. 

A  better  situation  would  have  been  to  interview  commercial  companies  from  various 
industries.  This  research  was  focused  on  companies  or  organizations  in  the  aerospace 
industry.  Interviews  with  a  depot  buying  agency  and  an  operational  unit  would  also  have 
strengthened  my  research. 

Lastly,  many  of  the  respondents  were  asked  open-ended  questions  with  no  time  to 
prepare.  Some  of  the  information  recorded  during  the  interviews  would  have  been  more 
accurate  had  the  respondents  been  sent  the  questions  prior  to  the  interviews.  This  was 
done  to  try  and  get  the  most  honest  answers  from  those  participating. 

Recommendations  for  Future  Research 

Future  research  should  include  a  comparison  of  the  actual  scorecards  that  companies 
have  utilized.  In  this  study  I  received  two  scorecards,  one  from  FedEx  and  one  from 
UTC.  If  the  USAF  is  going  to  implement  scorecards  it  should  obtain  feedback  on  the 
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most  and  least  desirable  rating  categories.  Another  recommendation  would  be  to  sample 
the  suppliers  that  have  scorecards  on  file  with  these  companies.  Although  the  companies 
felt  that  the  suppliers  welcome  this  type  of  rating  system,  it  would  be  interesting  to  know 
if  this  holds  true.  Additionally,  supplier  qualification  and  certification  plays  a  huge  factor 
in  establishing  long-term  relationships.  The  USAF  is  currently  investigating  ways  to 
recognize  superior  performers  without  it  resulting  in  favoritism  for  future  acquisitions. 
Lockheed  Martin’s  STAR  supplier  program  seems  to  be  an  effective  way  of  building 
pride  in  workmanship  for  its  suppliers.  Further  research  could  focus  on  the  “best 
practices”  of  various  supplier  award  programs. 

Based  upon  the  results  of  this  research  effort,  there  is  evidence  to  support  the 
hypotheses  that  automation  can  be  a  positive  enhancement  to  supplier  performance 
evaluations.  If  this  is  implemented  with  the  USAF  community  training  will  be  required 
at  the  earliest  stages  of  training  for  purchasing  personnel.  Familiarity  and  buy-in  with  the 
system  will  be  critical  in  determining  its  success  or  failure  (Boeing). 

Additional  Recommendations 

Many  commercial  firms  have  establishing  sourcing  departments  within  their 
company.  These  departments  are  responsible  for  all  major  source  selections.  Based  on 
the  lack  of  experience  that  was  discussed  in  the  C-130  AMP  Interview,  I  propose  that  a 
source  selection  office  at  a  product  center  be  more  than  just  a  support  facility.  This 
organization  could  become  a  “sourcing  team”.  Theoretically,  the  same  individuals  that 
work  the  past  performance  portion  for  the  F-16  avionics  upgrade  would  work  the  past 
performance  portion  of  the  B-2  engine  upgrade 
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Many  suppliers  in  the  automotive  industry  are  held  to  the  same  process/quality 
standards  as  aerospace  firms.  Honeywell  mentioned  that  many  of  these  suppliers  are  no 
longer  doing  business  with  Ford  or  GM  because  of  optimization  (i.e.  supplier  reductions). 
These  suppliers  have  many  of  the  capabilities  and  resources  that  could  be  applied  towards 
DoD  efforts.  It  would  be  interesting  to  see  if  these  companies  could  help  alleviate  some 
of  the  problems  associated  with  the  DMS  (Diminishing  Manufacturing  Sources) 
challenge  that  the  DoD  currently  faces. 
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Attachment  1 


Note:  Open  Ratings  calculates  supplier  performance  scores  using  a  sophisticated 
algorithm  that  takes  into  account  the  amount  of  information  available  on  a  supplier,  the 
recency  of  the  information,  and  the  accuracy  of  the  raters.  Ratings  range  from  0  to  100, 
however,  this  is  not  a  percentile  score. 

Summary  Performance 
Rating 
92 

Indicative  of  likely  overall  performance 

SIC  Level  Quintile 

Bottom  Top 

SIC:  8742/Management  Consulting 
Services 


Detailed  Performance  Ratings 
RELIABILITY: 

How  reliably  do  you  think  this  company  follows  through 
on  its  commitments? 

92 

COST: 

How  closely  did  your  final  total  costs  correspond  to 
your  expectations  at  the  beginning  of  the  transaction? 

92 

ORDER  ACCURACY: 

How  well  do  you  think  the  product/service  delivered 
matched  your  order  specifications  and  quantity? 

92 

DELIVERY/TIMELINESS: 

How  satisfied  do  you  feel  about  the  timeliness 
of  the  product/service  delivery? 

93 

QUALITY: 

How  satisfied  do  you  feel  about  the  quality  of 
the  product/service  provided  by  this  company? 

94 

BUSINESS  RELATIONS: 

How  easy  do  you  think  this  company  is  to  do 
business  with? 

93 

PERSONNEL: 

How  satisfied  do  you  feel  about  the  attitude,  courtesy, 
and  professionalism  of  this  company's  staff? 

94 

CUSTOMER  SUPPORT: 

How  satisfied  do  you  feel  about  the  customer 
support  you  received  from  this  company? 

92 

RESPONSIVENESS: 

How  responsive  do  you  think  this  company  was  to 
information  requests,  issues,  or  problems  that  arose  in  the 
course  of  the  transaction? 

90 

Source:  (OpenRatings.com,  2002) 


SIC  Level  Quintile 

ooooo 

Bottom  |  |  ^“1  |  To; 

SIC:  8742/Management  Consult 

Services 

68 


Attachment  2 
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Appendix  A:  Interview  Questions  for  Contractor/Program  Office  Personnel 


Guidelines:  The  information  obtained  from  this  interview  will  be 
incorporated  into  the  thesis  without  mentioning  names  of  those  interviewed. 
Any  references  made  regarding  the  interviews  will  mention  the 
company/program  office  and  the  individual’s  job  title  (e.g.  contracting 
officer  or  program  manager).  Each  interviewee  must  also  give  verbal 
permission  for  any  use. 

Contractor 

Pre-Award  Evaluation 

1.  What  are  the  categories/factors  of  assessment  used  for  evaluating  past  performance? 
How  were  these  categories  constructed  (i.e.  industry  standard,  from  within)? 

2.  Is  your  rating  system  quantitative  (e.g.  late  delivery  in  terms  of  #  of  days)  or 
qualitative  (e.g.  quality  performance)? 

3.  How  many  days  on  average  does  it  take  you  to  award  a  contract  (e.g.  clock  starts 
when  a  requirement  or  funding  document  is  received)? 

4.  Has  your  pre-award  evaluation  process  changed  over  the  last  5  years? 

5.  Have  you  been  able  to  quantity  any  savings  over  the  last  5  years  with  your  evaluation 
process? 

1C.  What  is  your  impression  of  the  current  DoD  past  performance  and  risk  assessment? 

6.  Does  your  company  receive  feedback  from  other  business  partners  concerning  your 
own  past  performance  evaluation  (e.g.  how  you  are  perceived  or  rate  with  other  firms)? 

7.  Would  you  categorize  most  of  your  contracts  fixed-price,  incentive  based,  or  cost- 
plus? 

8.  Is  the  importance  of  price  or  financial  performance  tailored  for  each  acquisition? 

9.  Is  price  the  most  important  factor  in  evaluation? 

Post- Award  Evaluation 

1 .  How  is  post-award  evaluation  maintained  by  your  firm? 

2.  Is  the  process  automated  or  manual?  What  is  the  role  of  purchasing  personnel  in  this 
process? 
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2C.  What  is  your  impression  of  third-party  rating  systems  (e.g.  OpenRatings)? 

3C.  If  your  process  is  automated,  who  designed  the  software  or  what  third-party 
company  maintains  this  database? 

3 .  How  much  time  (hours,  days)  is  required  to  complete  a  post-award  evaluation?  How 
often  is  it  performed  (never,  monthly,  quarterly)? 

4.  Does  your  evaluation  after  award  affect  award-fees  or  bonuses  given  for  above 
average  performance? 

5.  What  factors  do  you  perceive  will  affect  the  success  or  failure  of  a  contract? 
Examples  include:  Quality  Management  Practices,  Documentation  and  self-audit, 
process/manufacturing  capability,  management  of  the  firm,  design  and  development 
capabilities,  and  cost  reduction  capabilities 

General  Questions 

1.  How  would  you  describe/define  strategic  sourcing?  Does  it  apply  before  and  after 
award? 

4C.  Do  you  maintain  one  rating  for  each  supplier  which  covers  all  contracts  or  are  they 
given  a  rating  which  is  based  on  their  performance  on  one  particular  contract? 

2.  What  3  things  could  you  have  done  better  in  pre-award  and  post- award  evaluations 
knowing  what  you  know  now? 

3.  How  much  training  is  given  in-house  regarding  supplier  evaluations? 

4.  Does  your  firm  use  multi-functional  teams  when  performing  pre-award  and  post¬ 
award  evaluations  or  do  different  departments  (e.g.  engineering,  purchasing,  marketing) 
act  separately? 

5.  How  many  individuals  typically  participate  in  a  pre-award  or  post- award  evaluation 
(i.e.  $1M,  $50M,  &  $100M+  acquisitions)? 

6.  Would  you  characterize  you  relationships  with  suppliers  short-term  or  long-tenn? 

7.  Has  your  relationship  with  your  suppliers  improved  or  suffered  because  of  your 
evaluation  procedures? 

5C.  Does  your  company  do  business  in  the  aircraft  or  manufacturing  industries? 

6C.  What  regulations  does  your  company  have  which  outline  the  pre-award  evaluation 
process? 
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USAF  Program  Office  Personnel 

Pre- Award  Evaluation 

1 .  What  are  the  categories  of  assessment  used  for  evaluating  past  performance?  How 
were  these  categories  constructed  (i.e.  USAF/AFMC  Guide,  DoD  Past  Perfonnance 
Guide,  from  within)? 

2.  Is  your  rating  system  quantitative  (e.g.  late  delivery  in  terms  of  #  of  days)  or 
qualitative  (e.g.  quality  performance,  technical  performance)? 

3.  How  many  days  on  average  does  it  take  you  to  award  a  contract  (e.g.  clock  starts 
when  a  requirement  or  funding  document  is  received)? 

4.  Has  your  pre-award  evaluation  process  changed  over  the  last  5  years? 

5.  Have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your  evaluation 
process? 

1G.  What  is  your  perception  of  the  FAR’s  restrictions  on  the  pre-award  evaluation 
process? 

6.  Does  your  office  receive  feedback  from  suppliers  regarding  past  perfonnance 
evaluation  (e.g.  source  selections)? 

2G.  How  many  protests  have  you  received? 

7.  Would  you  categorize  most  of  your  contracts  fixed-price,  incentive  based,  or  cost- 
plus? 

8.  Is  the  importance  of  price  or  financial  performance  tailored  for  each  acquisition? 

9.  Is  price  the  most  important  factor  in  evaluation? 

Post- Award  Evaluation 

1 .  How  is  post-award  evaluation  maintained  by  the  program  office?  Do  you  use  USAF 
or  AFMC  guides? 

2.  What  is  the  role  of  purchasing  personnel  vs.  program  managers  in  this  process? 

3G.  If  this  process  could  be  automated  would  it  alter  your  perception  of  the  DoD 
evaluation  process? 
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3.  How  much  time  (hours,  days,  months)  is  required  to  complete  a  post-award 
evaluation?  How  often  is  it  perfonned  (never,  monthly,  quarterly)? 

4.  Does  your  evaluation  allow  for  incentives  if  the  contractor  exceeds  the  requirement? 

5.  What  factors  do  you  perceive  will  affect  the  success  or  failure  of  a  contract? 

Examples  include:  Quality  Management  Practices,  Documentation  and  self-audit, 
process/manufacturing  capability,  management  of  the  firm,  design  and  development 
capabilities,  and  cost  reduction  capabilities 

4G.  Does  the  program  office  currently  maintain  a  performance  file  for  each  of  its 
contractors? 

General  Questions 

1.  How  would  you  describe/define  strategic  sourcing?  Does  it  apply  before  and  after 
award? 

2.  What  3  things  could  you  have  done  better  in  pre-award  and  post- award  evaluations 
knowing  what  you  know  now? 

3.  How  much  training  is  given  in-house  regarding  supplier  evaluations? 

4.  Does  your  program  office  use  multi-functional  teams  when  perfonning  pre-award  and 
post-award  evaluations  or  do  different  functionals  (e.g.  engineering,  purchasing, 
marketing)  act  separately? 

5.  How  many  individuals  typically  participate  in  a  pre-award  or  post-award  evaluation 
(i.e.  $1M,  $50M,  &  $100M+  acquisitions)? 

6.  Would  you  characterize  you  relationships  with  suppliers  short-term  or  long-tenn? 

7.  Has  your  relationship  with  your  suppliers  improved  or  suffered  because  of  your 
evaluation  procedures? 
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Appendix  B:  Transcribed  Interviews  (Delta  Airlines) 


Interview  Questions  for  Contractor/Program  Office  Personnel 


Contractor 

Pre- Award  Evaluation 

1.  What  are  the  categories/factors  of  assessment  used  for  evaluating  past  performance? 
How  were  these  categories  constructed  (i.e.  industry  standard,  from  within)?  We  are 
talking  about  On-time  Delivery,  Lead-Times,  Price  history,  Quality  (incoming  rejects), 
internal  responsiveness  (surveys)  of  the  supplier.  They  were  constructed  within.  We 
discussed  this  with  suppliers  and  we  use  standard  systems  but  it  is  probably  common  with 
other  companies.  Other  than  American  Airlines  we  think  we  are  the  most 
comprehensive. 

2.  Is  your  rating  system  quantitative  (e.g.  late  delivery  in  terms  of  #  of  days)  or 
qualitative  (e.g.  quality  performance)?  Quantitative  as  much  as  possible.  I’ll  give  the 
delivery  a  score  based  on  a  spreadsheet.  We  have  exceptions  if  something  is  late.  Very 
quantitative. 

3.  How  many  days  on  average  does  it  take  you  to  award  a  contract  (e.g.  clock  starts 
when  a  requirement  or  funding  document  is  received)?  Let’s  take  a  case,  the  part  is 
already  designed  by  the  OEM.  I  know  the  quantity.  We  might  bring  in  a  part  on  an 
evaluation  basis.  We  might  visit  the  supplier.  The  process  could  take  3  weeks  but  we 
kind  of  spread  it  out.  It  also  depends  on  how  urgent  the  need  is. 

4.  Has  your  pre-award  evaluation  process  changed  over  the  last  5  years?  I’ve  only  been 
here  3  and  a  half  years,  but  not  so  much  from  a  pre-award  standpoint.  We  give  suppliers 
on-going  feedback.  We’ve  done  some  automation  on  how  we  collect  and  distribute 
information  to  supplier.  That’s  been  enhanced.  More  feedback  to  suppliers. 

5.  Have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your  evaluation 
process?  We  went  through  a  10%  savings  on  on-time  delivery.  We  measured  that  over  a 
three  year  period.  However,  it’s  hard  to  put  dollars  to  that.  Most  orders  are  for  buying 
materials  for  stock.  I  wouldn’t  want  to  put  a  dollar  figure  on  that.  Not  that  I  haven’t 
tried. 

1C.  What  is  your  impression  of  the  current  DoD  past  performance  and  risk  assessment? 
My  impression  is  that  DoD  does  a  very  thorough  job.  It  will  take  a  lot  of  time  and  it  will 
cost  a  lot.  But  it  is  upfront  work.  In  contrast  with  Delta  Airlines  is  that  we  should  do 
more  upfront  work. 

6.  Does  your  company  receive  feedback  from  other  business  partners  concerning  your 
own  past  performance  evaluation  (e.g.  how  you  are  perceived  or  rate  with  other  firms)?  I 
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get  feedback  informally.  We  don’t  have  a  formal  mechanism  for  doing  that.  We’ve 
talked  about  that.  We  have  trained  suppliers  on  how  we’re  going  to  measure  them  and 
the  provide  a  little  feedback.  Our  system  is  very  complex.  More  complicated  than  you 
think  it  would  be.  Some  suppliers  take  the  time  to  understand  our  system  and  others 
don’t. 

7.  Would  you  categorizes  most  of  your  contracts  fixed-price,  incentive  based,  or  cost- 
plus?  Mostly  fixed-price. 

8.  Is  the  importance  of  price  or  financial  performance  tailored  for  each  acquisition? 

It  depends  on  the  complexity  of  what  we’re  buying.  If  we’re  buying  a  heads  up  display 
or  electronics  you’re  buying  a  design.  If  you’re  buying  a  design  then  price  becomes  less 
important.  If  you’re  buying  a  part  then  price  is  most  important. 

9.  Is  price  the  most  important  factor  in  evaluation?  It  is  a  factor  but  not  most  important 
in  every  acquisition. 

Post- Award  Evaluation 

1 .  How  is  post-award  evaluation  maintained  by  your  firm?  We  have  a  scorecard  process 
that  we  give  to  our  top  50  or  so  suppliers  in  technical  operations.  We  weigh  them  in  on 
the  categories  discussed  in  the  pre-award  phase.  We  also  have  quarterly  meetings  with 
suppliers  to  talk  about  perfonnance  and  what  we  need  in  the  future. 

2.  Is  the  process  automated  or  manual?  What  is  the  role  of  purchasing  personnel  in  this 
process?  Through  a  fairly  automated  means  I  extract  data  out  of  a  database.  I  assess  on- 
time  delivery,  lead  time,  pricing,  and  one  for  quality.  These  are  four  spreadsheets.  The 
buyer  may  or  may  not  share  this  with  the  supplier  before  they  score.  We  use  an  access 
database.  The  buyer  will  then  create  one  spreadsheet  out  of  this.  The  buyer  has  the 
ability  to  change  that  if  they  want.  It  is  homemade  programming  and  a  lot  of 
maintenance  involved  in  keeping  this  system  going. 

2C.  What  is  your  impression  of  third-party  rating  systems  (e.g.  OpenRatings)?  I’m 
familiar  with  that.  It  provides  a  useful  data  point  but  you  wouldn’t  want  to  put  to  much 
confidence  on  it.  We  have  been  considering  third-party  systems  and  we  decided  not  to 
use  them.  We  are  looking  at  a  web-based  system  instead.  I  was  out  at  Boeing  last  month 
and  I  understand  that  they  are  using  something  similar.  With  Boeing,  they  see 
themselves  as  a  business  that  can  supply  information  to  the  airlines.  They’re  telling  me 
how  Rockwell  Collins  is  doing,  but  I  know  how  Rockwell  Collins  is  doing.  In  the  end, 
Boeing  is  not  really  in  a  position  to  inform  Delta  how  one  of  our  suppliers  is  doing. 

3C.  If  your  process  is  automated,  who  designed  the  software  or  what  third-party 
company  maintains  this  database?  The  system  is  homemade  and  was  designed  in-house. 

3.  How  much  time  (hours,  days)  is  required  to  complete  a  post-award  evaluation?  How 
often  is  it  perfonned  (never,  monthly,  quarterly)?  For  the  50  main  suppliers  it  takes  a 
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buyer  about  a  half  hour.  It  is  up  to  the  buyer  how  often  they  report  the  scorecard 
information  to  the  suppliers.  Some  suppliers  want  to  see  the  infonnation  at  the  first  of 
the  month  and  others  aren’t  to  concerned.  The  minimum  requirement  is  to  give  the 
suppliers  quarterly  data.  The  data  comes  in  monthly  buckets  but  is  rolled  up  at  the  end  of 
the  quarter. 

4.  Does  your  evaluation  after  award  affect  award-fees  or  bonuses  given  for  above 
average  performance?  They  can,  we  have  cases  where  assign  penalties  if  a  supplier 
doesn’t  meet  a  performance  guarantee.  We  have  at  time  set  up  penalties,  but  it  is  more 
the  exception  than  the  rule.  We  are  trying  to  do  more  of  this.  What  is  more  typical  in  the 
contract  is  that  the  supplier  promises  to  participate  in  the  supplier  perfonnance  program 
in  the  future  rather  than  penalize  them. 

5.  What  factors  do  you  perceive  will  affect  the  success  or  failure  of  a  contract?  Effective 
communication  of  needs. 

General  Questions 

1.  How  would  you  describe/define  strategic  sourcing?  Does  it  apply  before  and  after 
award?  Yes,  we  have  a  strategic  sourcing  process  here  at  Delta.  It  is  the  selection  of 
long-term  opportunities  for  sourcing  on  a  total  cost  basis.  We  are  trying  to  capture  this 
when  we  select  suppliers.  We  have  strategic  sourcing  department  and  we  work  with  a 
third-party  who  has  web-enabled  this  process.  It  is  a  seven  step  process  on  the  web  that 
steps  the  buyer  to  answer  questions.  It  provides  a  permanent  repository  for  results  and  in 
theory  future  buyers  can  leverage  off  of  this. 

4C.  Do  you  maintain  one  rating  for  each  supplier  which  covers  all  contracts  or  are  they 
given  a  rating  which  is  based  on  their  performance  on  one  particular  contract?  It  is  across 
the  board. 

2.  What  3  things  could  you  have  done  better  in  pre-award  and  post- award  evaluations 
knowing  what  you  know  now?  We  have  to  work  harder  at  having  objective  decision 
criteria  up-front.  We  talk  about  what  we’re  trying  to  achieve  but  often  we  don’t  nail 
down  the  objective  terms.  We  often  select  a  supplier  based  on  what  feels  right. 

3.  How  much  training  is  given  in-house  regarding  supplier  evaluations?  Not  very  much 
at  all.  We  have  courses  that  are  made  available.  A  year  or  two  back  we  wanted  all  the 
buyers  to  become  certified  purchasing  managers.  They  went  to  six  or  seven  days  of 
training  that  covered  a  little  bit  of  this. 

4.  Does  your  firm  use  multi-functional  teams  when  performing  pre-award  and  post¬ 
award  evaluations  or  do  different  departments  (e.g.  engineering,  purchasing,  marketing) 
act  separately?  They’re  not  likely  to  be  physically  located  in  the  same  office,  but  we  do 
use  multi-disciplinary  teams.  There  tends  to  be  supply  chain  leader  and  a  business  team 
leader.  They  draw  in  resources  from  other  groups  as  needed.  The  supply  chain  guy  is  the 
only  full  time  guy. 
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5.  How  many  individuals  typically  participate  in  a  pre-award  or  post- award  evaluation 
(i.e.  $1M,  $50M,  &  $100M+  acquisitions)?  For  a  $50M  purchase  that  is  pretty  large.  It’s 
hard  to  say.  20  or  30  people  wouldn’t  surprise  me.  I  do  $70M  to  $80M  spend  with 
Boeing.  The  designs  are  pretty  fixed.  There’s  probably  a  core  of  6  to  10  people  with  this 
type  of  buy. 

6.  Would  you  characterize  you  relationships  with  suppliers  short-term  or  long-term? 
Long-term. 

7.  Has  your  relationship  with  your  suppliers  improved  or  suffered  because  of  your 
evaluation  procedures?  I  can’t  answer  that.  We’ve  started  to  in-sourcing  over  the  last 
three  years.  Now,  we’re  competitors  with  some  of  our  old  suppliers.  Delta  has  pulled 
more  stuff  back  in  over  the  last  couple  of  year. 

5C.  Does  your  company  do  business  in  the  aircraft  or  manufacturing  industries?  Yes 

6C.  What  regulations  does  your  company  have  which  outline  the  pre-award  evaluation 
process?  We  have  worked  to  establish  some  procedures.  They  are  policy  level 
guidelines  on  how  we  conduct  business.  Most  of  the  business  in  technical  operations  are 
standard.  In  supply  chain  we  really  don’t  have  standard  procedures.  We  have  to  be 
careful  to  make  sure  we  are  using  FAA  approved  suppliers.  There  is  a  set  of  financial 
policies  in  supply  chain. 
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(Honeywell) 


Interview  Questions  for  Contractor/Program  Office  Personnel 


Contractor 

Pre- Award  Evaluation 

1.  What  are  the  categories/factors  of  assessment  used  for  evaluating  past  performance? 
How  were  these  categories  constructed  (i.e.  industry  standard,  from  within)?  If  it’s  a 
brand  new  supplier,  we’re  not  going  to  have  actual  perfonnance  values  for  them.  We 
won’t  know  what  their  supplier  quality  perfonnance,  on-time  delivery  has  been.  We’ll 
start  off  with  a  telephone  interview.  This  is  a  general  process,  and  varies  from  site  to  site. 
Number  one  we’ll  determine  who  they’ve  done  business  with.  We  use  open  ratings  to 
look  the  supplier  up  and  look  at  their  predictive  perfonnance  in  the  area  of  financial  risk 
and  overall  business  performance.  We  look  to  see  if  that  supplier  is  in  poor  shape 
financially  or  not  performing  well  amongst  their  peers.  Then  I’ll  ask  the  supplier  if 
they’ve  been  registered  or  certified  to  be  compliant  with  certain  aerospace  quality 
standards.  All  Honeywell  aerospace  have  adopted  the  America’s  Aerospace  Quality 
Group  and  the  International  Association  of  Aerospace  Quality  Group  standards.  We  have 
also  used  the  FAA  and  DoD’s  quality  standards.  We  are  looking  for  AS  9100  and 
AS9120  if  they’re  a  distributor.  Or  we’ll  look  at  AS9003  if  they’re  a  manufacturer  of 
simple  parts  or  a  distributor.  We  also  use  a  MP3  which  is  a  quick  plant  tour  path 
assessment.  Are  they  above  average,  average,  or  below  average.  The  supplier  will  in  a 
sense  do  a  self  assessment.  They  look  at  inventory  management  and  how  they  manage 
financial  systems.  If  you’re  an  international  supplier  we  ask  for  a  virtual  tour  of  their 
plant  with  a  digital  camera  to  take  pictures.  You  get  the  boat  load  if  you’re  building 
important  products  for  us.  If  it’s  a  simple  part  we’ll  scale  down. 

2.  Is  your  rating  system  quantitative  (e.g.  late  delivery  in  terms  of  #  of  days)  or 
qualitative  (e.g.  quality  performance)?  It’s  really  a  mix.  Twenty  percent  of  our  suppliers 
are  doing  business  with  other  divisions  of  our  company.  The  standards  such  as  number 
of  defects  will  vary  If  you  are  an  automotive  supplier  or  commercial  supplier.  For 
example,  a  casting  or  forging  supplier  will  be  hard  pressed  to  get  under  10,000  PPM  for 
aerospace.  It’s  hard  to  be  all  quantitative  with  quality  because  there  are  different 
industries. 

3.  How  many  days  on  average  does  it  take  you  to  award  a  contract  (e.g.  clock  starts 
when  a  requirement  or  funding  document  is  received)?  We’ve  had  regional  conferences 
for  international  companies  in  Mexico  to  develop  aerospace  companies  in  Mexico.  This 
takes  a  lot  of  upfront  work.  We’ve  offered  them  packages  to  quote  on.  This  process 
takes  about  6  months.  We  have  to  very  confident  that  global  companies  really 
understand  the  terms  and  conditions.  This  takes  a  little  longer. 
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4.  Has  your  pre-award  evaluation  process  changed  over  the  last  5  years?  Oh,  yes.  It’s 
much  more  in-depth.  We  have  accelerated  the  number  of  auctions.  Originally  we  invited 
suppliers  that  came  recommended.  We  found  that  many  suppliers  weren’t  capable  of 
meeting  the  requirement.  We  spend  a  lot  more  time  up-front. 

5.  Have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your  evaluation 
process?  Overall  we  are  shrinking  our  supply  base  and  improving  the  defect  level  and 
over  the  last  two  years  there’s  probably  been  a  25%  improvement  in  the  quality  level.  Is 
that  because  we’re  managing  suppliers  better  or  selecting  them  I’m  not  sure.  We’ve  done 
a  lot  of  stuff  simultaneously. 

1C.  What  is  your  impression  of  the  current  DoD  past  performance  and  risk  assessment? 
Not  directly.  I  have  listened  to  informants  speak  about  the  process.  Given  your  resource 
constraints,  I  like  the  process. 

6.  Does  your  company  receive  feedback  from  other  business  partners  concerning  your 
own  past  performance  evaluation  (e.g.  how  you  are  perceived  or  rate  with  other  firms)? 
FAA  in  their  audits  one  of  their  biggest  complaints  is  that  the  OEMs  are  not  managing 
their  suppliers  very  well.  A  year  or  so  ago,  they  took  Boeing  to  task.  So  Boeing  kicked 
us.  Historically,  we  have  tolerated  poor  performance  up  and  down  the  line.  But  we’re 
getting  healthy.  When  Boeing  comes  in  now,  people  are  much  more  responsive.  If  you 
want  to  do  business  with  them  we  must  comply. 

7.  Would  you  categorize  most  of  your  contracts  fixed-price,  incentive  based,  or  cost- 
plus?  Fixed-Price.  Boeing  expects  productivity  improvement  and  price  reductions. 

Some  people  also  think  that  Honeywell  is  pretty  mean.  After  9/11,  Boeing  said  they 
couldn’t  pay  us.  It  took  90  to  120  days  to  make  payment.  We  had  to  tell  our  suppliers 
the  same  thing. 

8.  Is  the  importance  of  price  or  financial  performance  tailored  for  each  acquisition? 
Bottom  line,  price  is  the  most  important.  If  a  supplier  has  their  act  together,  by 
minimizing  warranty  costs,  and  delivering  defect-free  material,  the  price  will  win. 

9.  Is  price  the  most  important  factor  in  evaluation?  Within  the  aerospace  business, 
quality  is  a  must.  Price  is  all  that  remains  to  discriminate  based  on  the  high  standards  of 
the  industry.  We  have  asked  our  suppliers  to  achieve  a  6%  reduction  in  their  price. 

Post- Award  Evaluation 

1 .  How  is  post-award  evaluation  maintained  by  your  firm?  With  new  products  we  are 
looking  for  management  capability,  project  plans,  schedule  targets,  process  capability, 
and  first-article  production.  If  we’re  buying  a  microprocessor  from  Intel  they  tell  you 
when  it  comes  to  market.  We  don’t  have  a  lot  of  leverage  with  Intel. 

2.  Is  the  process  automated  or  manual?  What  is  the  role  of  purchasing  personnel  in  this 
process?  We  use  Microsoft  Project  as  much  as  possible.  We  use  NetMeeting  and 
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electronic  media  to  communicate  with  suppliers.  We  measure  quality,  parts  rejects  over 
parts  received  and  on-time  delivery.  By  next  year  we  will  have  a  web-based  electronic 
scorecard.  On-Time  Delivery  is  5  days  early  or  5  days  late.  Productivity  is  kept-in 
house.  Most  of  the  procurement  people  don’t  want  the  supplier  to  know  the  productivity 
ratings.  We’re  half  way  through  a  global  system  that  will  show  us  spend  and  ratings 
information  by  part  number. 

2C.  What  is  your  impression  of  third-party  rating  systems  (e.g.  OpenRatings)?  We  use 
them  as  a  indicator  or  alert  mechanism.  If  I’m  a  commodity  manager  of  75  suppliers,  I 
don’t  have  time  to  look  at  financials.  OpenRatings  lets  me  set  up  a  watch  list  and  it  flags 
me.  If  I  see  that  a  supplier’s  perfonnance  against  their  peers  is  declining  I  can  plan 
accordingly.  We’re  using  it  as  a  early  warning  or  heads  up.  We  didn’t  buy  their 
scorecard  option.  United  Technologies  is  doing  this,  but  not  us. 

3C.  If  your  process  is  automated,  who  designed  the  software  or  what  third-party 
company  maintains  this  database?  Other  than  open  ratings,  the  evaluation  system 
(scorecard)  was  designed  in-house. 

3 .  How  much  time  (hours,  days)  is  required  to  complete  a  post-award  evaluation?  How 
often  is  it  perfonned  (never,  monthly,  quarterly)?  This  process  is  done  monthly, 
quarterly,  and  annually. 

4.  Does  your  evaluation  after  award  affect  award-fees  or  bonuses  given  for  above 
average  performance?  Our  suppliers  don’t  get  incentives  other  than  the  opportunity  to 
get  more  business  with  us.  We  do  ask  for  discounts  if  we  pay  suppliers  early. 

5.  What  factors  do  you  perceive  will  affect  the  success  or  failure  of  a  contract?  Number 
one  I  don’t  get  the  product  on-time  or  it  is  delivered  late,  Number  two  the  product  doesn’t 
meet  specifications,  and  number  three  I  don’t  meet  my  productivity  goals. 

General  Questions 

1 .  How  would  you  describe/define  strategic  sourcing?  Does  it  apply  before  and  after 
award?  Strategic  sourcing  means  leveraging  our  spend  to  get  the  best  price  from  our 
suppliers. 

4C.  Do  you  maintain  one  rating  for  each  supplier  which  covers  all  contracts  or  are  they 
given  a  rating  which  is  based  on  their  performance  on  one  particular  contract?  We  use 
report  actual  results  for  each  division.  By  2003  it  will  be  a  roll  up  to  the  corporate  level. 

2.  What  3  things  could  you  have  done  better  in  pre-award  and  post- award  evaluations 
knowing  what  you  know  now?  There  have  been  times  where  a  supplier  does  a  self 
assessment  which  shows  they’re  capable  but  in  reality  they  can’t  meet  the  requirement. 

In  a  way  we’ve  often  mislead  the  supplier. 
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3.  How  much  training  is  given  in-house  regarding  supplier  evaluations?  Honeywell 
requires  40  hours  every  year  of  training  for  everyone.  What  we’re  trying  to  do  now  is 
build  the  training  into  the  business  processes  and  into  the  curriculum  for  procurement 
personnel.  We  want  the  purchasing  personnel  to  understand  the  preferred  steps  for 
selecting  a  supplier.  We  use  what  we  call  OneSource  to  drive  the  procurement.  Our 
training  is  more  process  than  specific  functional  training  now. 

4.  Does  your  firm  use  multi-functional  teams  when  performing  pre-award  and  post¬ 
award  evaluations  or  do  different  departments  (e.g.  engineering,  purchasing,  marketing) 
act  separately?  We  use  integrated  product  teams.  It  depends  on  the  complexity  of  the 
product.  If  it  is  a  major  procurement  the  teams  sits  down  and  plans  out  a  strategy.  For 
simple  purchases,  purchasing  and  quality  personnel  might  not  have  to  discuss  the  matter. 
Most  of  the  team  members  do  not  sit  in  the  same  work  area  and  often  are  not  in  the  same 
building  or  the  same  state.  We  also  have  commodity  managers  that  are  considered 
specialist  for  particular  items. 

5.  How  many  individuals  typically  participate  in  a  pre-award  or  post- award  evaluation 
(i.e.  $1M,  $50M,  &  $100M+  acquisitions)?  Right  now,  a  $50  would  be  a  strategic 
procurement  and  we  would  sign  a  long-term  contract.  A  commodity  lead,  a  quality  rep, 
and  engineering  rep.  There  will  probably  be  4  key  people  but  it  will  touch  a  lot  more 
people. 

6.  Would  you  characterize  you  relationships  with  suppliers  short-tenn  or  long-term? 
Long-Term 

7.  Has  your  relationship  with  your  suppliers  improved  or  suffered  because  of  your 
evaluation  procedures?  The  suppliers  probably  think  on  some  levels  that  it  has  improved. 
Now,  have  we  gotten  any  easier  to  work  with,  probably  not.  We’re  still  very  demanding 
and  very  impatient  when  it  comes  to  not  meeting  objectives.  We  expect  our  suppliers  to 
be  agile  and  to  share  more  of  the  costs.  We  expect  them  to  help  us  manage  the  inventory. 
We’ve  significantly  increased  the  responsibility  that  we’ve  placed  on  the  suppliers. 

5C.  Does  your  company  do  business  in  the  aircraft  or  manufacturing  industries?  Yes 

6C.  What  regulations  does  your  company  have  which  outline  the  pre-award  evaluation 
process?  There  is  command  media  at  each  one  of  the  sites/divisions  that  would  describe 
the  role  of  the  quality  personnel.  At  the  strategic  level  we  have  common  standards. 

There  is  general  business  guidelines  and  electronic  training  procedures. 


81 


(Raytheon) 


Interview  Questions  for  Contractor/Program  Office  Personnel 


Contractor 

Pre- Award  Evaluation 

1.  What  are  the  categories/factors  of  assessment  used  for  evaluating  past  performance? 
How  were  these  categories  constructed  (i.e.  industry  standard,  from  within)?  We  look  at 
on-time  delivery  internally  to  our  own  needs,  quality  in  number  of  lots  received  vs. 
rejected,  and  a  competitive  benchmark.  It  is  schedule,  quality,  and  price.  These  are 
universal  categories  and  an  industry  standard.  On  the  direct  side  we  have  qualified 
suppliers  where  we  only  solicit  those  that  are  capable.  I  deal  with  LockheedMartin, 
Boeing,  and  UAE  and  our  categories  are  pretty  much  the  same. 

2.  Is  your  rating  system  quantitative  (e.g.  late  delivery  in  terms  of  #  of  days)  or 
qualitative  (e.g.  quality  performance)?  It’s  both  quantitative  and  qualitative.  We  try  to 
be  factual  but  there  are  times  when  we  must  be  subjective.  When  there  is  a  SCAR 
(supplier  corrective  action  report)  we  will  work  with  the  supplier  to  get  them  back  up  to 
speed.  An  example  of  a  quantitative  rating  is  number  of  lots  vs.  number  of  items 
rejected. 

3.  How  many  days  on  average  does  it  take  you  to  award  a  contract  (e.g.  clock  starts 
when  a  requirement  or  funding  document  is  received)?  For  a  $50M  buy  it  usually  takes 
us  60  to  90  days.  We  will  fact  find  and  do  cost  and  pricing  data. 

4.  Has  your  pre-award  evaluation  process  changed  over  the  last  5  years?  We’ve  gotten 
much  more  electronic  and  web-enabled.  It’s  changed  from  a  tool  and  process  standpoint. 
We  have  a  universal  rating  which  is  input  and  kept  in  a  database  to  roll  things  up  and  can 
be  seen  across  the  company. 

5.  Have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your  evaluation 
process?  Speed,  Agility  are  inherent.  We’ve  certainly  been  able  to  deploy  resources 
more  strategically  than  tactically  in  terms  of  order  placement.  We  can  make  decisions 
more  quickly.  I  can’t  give  you  exact  savings  but  I  can  say  speed  has  been  enhanced.  It’s 
more  flexible  when  automated. 

1C.  What  is  your  impression  of  the  current  DoD  past  performance  and  risk  assessment? 

I  think  its  fairly  consistent  with  what  we  do.  We  have  tried  to  mirror  our  process  to 
match  DoDs  in  some  ways. 

6.  Does  your  company  receive  feedback  from  other  business  partners  concerning  your 
own  past  performance  evaluation  (e.g.  how  you  are  perceived  or  rate  with  other  firms)? 
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When  we  talk  to  our  suppliers,  we  try  to  talk  to  our  key  suppliers  monthly.  We  try  to 
benchmark  them  against  their  competition  without  them  knowing  who  their  competition 
is. 


7.  Would  you  categorize  most  of  your  contracts  fixed-price,  incentive  based,  or  cost- 
plus?  50%  cost-plus  and  50%  fixed-price.  Stable  production  contracts  are  fixed  and 
development  is  cost-type. 

8.  Is  the  importance  of  price  or  financial  performance  tailored  for  each  acquisition? 

Yes  to  a  degree.  It’s  not  a  scientific  formula.  We  look  at  the  prime  proposal  and  the 
strategic  suppliers.  I  don’t  think  we  have  a  lot  of  movement  in  how  we  do  it.  It’s  a 
30/30/30  with  a  10%  management  factor.  30%  quality,  30%  on-time  delivery,  and  30% 
price.  If  we  know  it’s  a  price  game  we’ll  sharpen  the  pencil. 

9.  Is  price  the  most  important  factor  in  evaluation?  I  don’t  think  it’s  the  most  important. 
Historically,  it  has  been  pegged,  however  it’s  changed.  We  really  try  and  emphasize  best 
value.  We’re  trying  to  communicate  in  any  forum  to  our  supply  base.  If  I  pay  10  cents 
less  and  I  end  up  having  to  ship  it  back  it  becomes  more  expensive. 

Post- Award  Evaluation 

1 .  How  is  post-award  evaluation  maintained  by  your  firm?  This  is  pretty  new.  We’ve 
done  some  reorganization.  The  supplier  ratings  system  is  now  being  proliferated.  If  a 
contractor  slips  below  a  threshold  of  perfonnance  we  shut  them  off.  We  put  them  on  a 
probationary  status.  Our  rating  system  is  called  SRS  (supplier  rating  system). 

2.  Is  the  process  automated  or  manual?  What  is  the  role  of  purchasing  personnel  in  this 
process?  Basically  what  happens  is  the  process  is  automated.  There  is  an  incoming 
inspection  loop  or  database  that  scores  the  supplier’s  perfonnance.  Once  a  product  is 
shipped  if  it  doesn’t  get  rejected  and  if  it  is  within  a  5  day  window  then  they  are  in 
compliance.  For  example,  we  don’t  pay  a  supplier  if  we  receive  an  item  more  than  5 
days  before  the  due  date.  We  are  trying  to  maintain  an  inventory  balance.  The  buyer 
does  a  query  where  the  punch  in  the  supplier  code  and  it  will  download  what  they  need 
for  perfonnance  data.  It  can  go  back  6  months,  1  year,  or  even  2  years. 

2C.  What  is  your  impression  of  third-party  rating  systems  (e.g.  OpenRatings)?  I’ve 
heard  of  them.  I  heard  it’s  pretty  good.  My  reaction  is  that  we’re  pretty  big  and  have  a 
lot  of  information  technology  in-house.  It  looks  attractive  but  as  upgrades  become 
necessary  it  can  become  expensive.  There’s  a  lot  to  control.  Even  though  they’re  a 
partner  the  third-party  is  in  it  to  make  money.  We  try  to  do  a  lot  of  things  ourself  rather 
than  turn  it  over  to  someone  else. 

3C.  If  your  process  is  automated,  who  designed  the  software  or  what  third-party 
company  maintains  this  database?  We  designed  it  ourselves  but  we  had  a  little  bit  of  help 
from  PWC.  They  helped  a  lot. 
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3.  How  much  time  (hours,  days)  is  required  to  complete  a  post-award  evaluation?  How 
often  is  it  performed  (never,  monthly,  quarterly)?  The  process  is  semi-automatic.  We 
have  data  people  that  load  the  infonnation  on  a  server.  The  buyers  look  at  quality  and 
delivery.  This  is  done  on  a  quarterly  basis. 

4.  Does  your  evaluation  after  award  affect  award-fees  or  bonuses  given  for  above 
average  performance?  If  it’s  specifically  tied  to  an  award  fee  contract.  Historical 
performance  doesn’t  really  matter,  but  if  we  tied  it  to  a  contract  and  they  achieved  a 
certain  threshold  they  would  be  paid. 

5.  What  factors  do  you  perceive  will  affect  the  success  or  failure  of  a  contract?  Early 
involvement  by  the  supply  chain  is  extremely  important.  It  allows  us  to  do  risk 
mitigations.  We  have  a  MPP  (Material  Program  Plan)  that  allows  us  to  look  at 
obsolescence.  The  biggest  factor  in  risk  is  getting  suppliers  involved  early.  In  the 
production  area  a  problem  is  asking  a  supplier  to  produce  at  too  fast  a  rate  or  asking  for  a 
difficult  delivery  date. 

General  Questions 

1.  How  would  you  describe/define  strategic  sourcing?  Does  it  apply  before  and  after 
award?  Strategic  sourcing  is  clearly  the  make  or  break  part  of  a  program.  I’ve  got 
groups  here  that  go  buy  material  for  a  number  of  different  applications.  The  whole 
element  of  strategy  is  getting  a  differentiator  among  suppliers.  The  folks  that  I  buy  the 
most  from  and  are  make  a  difference  should  be  brought  in  early.  We  want  to  partner  or 
team  with  these  folks.  Additionally,  I’ve  got  to  get  the  right  skill  set  internally.  I  need 
the  right  people  in-house.  The  key  to  our  operation  is  understanding  who  exactly  a 
strategic  supplier  is.  Who  should  we  link  with  is  very  important. 

4C.  Do  you  maintain  one  rating  for  each  supplier  which  covers  all  contracts  or  are  they 
given  a  rating  which  is  based  on  their  performance  on  one  particular  contract?  We  have 
both.  We  have  a  composite  that  rolls  up  total  performance.  We  then  have  a  menu  that 
gives  us  performance  on  a  particular  contract.  Our  chairmen  loves  it  because  he  was  very 
interested  to  know  how  he  was  doing  when  he  was  a  supplier.  He  wants  us  to  give  this 
information  to  our  supplier. 

2.  What  3  things  could  you  have  done  better  in  pre-award  and  post-award  evaluations 
knowing  what  you  know  now?  We  always  worry  about  solvency,  viability  financially, 
and  we  are  worried  ethics.  Ethics  is  becoming  significant  in  pre-award  and  after-award. 
We  have  a  material  program  manager  role  to  link  the  program  office  and  the  supplier. 
And  again,  early  involvement  to  find  out  if  the  supplier  can  meet  the  specification  or  get 
all  the  parts  they’ll  need. 

3.  How  much  training  is  given  in-house  regarding  supplier  evaluations?  We  have  pretty 
focused  deployment  teams  of  subject  matter  expects  (SME)  that  went  out  to  deploy  the 
system.  We  used  quality,  engineering,  and  supply  chain  personnel  to  get  the  word  out.  It 
helps  the  buyer  to  have  tools  to  assess  performance.  We  spent  2  days  of  training  (16 
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hours)  so  that  the  personnel  understand  the  integrity  of  the  data.  We  use  an  access 
database. 

4.  Does  your  firm  use  multi-functional  teams  when  performing  pre-award  and  post¬ 
award  evaluations  or  do  different  departments  (e.g.  engineering,  purchasing,  marketing) 
act  separately?  We  call  it  an  IPT.  The  bigger  programs  (DDX)  naval  program  of  a 
stealth  ship  with  munitions  on  it,  the  IPT  sit  in  the  same  building.  Other  programs  have  a 
program  office,  there  might  be  a  supply  chain  person  that  reports  out  of  their  home  office. 
The  best  situation  is  one  you  sit  people  together.  The  decision  making  is  real-time. 
There’s  no  replacement  for  having  people  co-located. 

5.  How  many  individuals  typically  participate  in  a  pre-award  or  post-award  evaluation 
(i.e.  $1M,  $50M,  &  $100M+  acquisitions)?  I  would  probably  say  15  people.  You’d  have 
supply  chain,  finance,  legal,  program  office,  quality,  field  folks  located  near  suppliers  to 
conduct  a  capability  survey,  and  engineering.  Probably  between  12  and  15  people. 

6.  Would  you  characterize  you  relationships  with  suppliers  short-tenn  or  long-term?  I 
think  they’re  long-term.  We’re  trying  to  rationalize  and  characterize  our  supply  base. 

We  do  business  with  too  many  folks.  We  have  a  cool  tool  called  spend  analysis.  We 
know  who  we  spend  money  with.  We  know  by  program  code  and  commodities  where 
we  spend.  I  think  the  way  to  get  better  long-terms  relationships  is  to  get  people  the  tools 
that  steer  them  towards  the  agreements.  PWC  helped  us  with  the  SAT  (spend  analysis 
tool).  If  we’re  bidding  on  a  job  in  Trent  Lott’s  district  we  can  drill  down  by  zip  code  and 
figure  out  who  we  buy  stuff  from  and  how  much.  We  can  report  how  much  money  will 
be  spend  in  that  district. 

7.  Has  your  relationship  with  your  suppliers  improved  or  suffered  because  of  your 
evaluation  procedures?  Very  much  so.  There’s  a  dispute  avenue  if  suppliers  object  to  the 
way  we  rate  them.  It  heightens  the  awareness  beyond  just  cost. 

5C.  Does  your  company  do  business  in  the  aircraft  or  manufacturing  industries?  Yes 

6C.  What  regulations  does  your  company  have  which  outline  the  pre-award  evaluation 
process?  Because  it  is  web-based  there  isn’t  a  ton  of  procedures  around  it.  It’s  an 
internal  document  that  is  published  or  e-mail.  It’s  pretty  much  navigation.  They  have  a 
desktop  guide  that  is  10  pages.  Initial  training  is  pretty  focused. 
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(Boeing) 


Interview  Questions  for  Contractor/Program  Office  Personnel 


Contractor 

Pre- Award  Evaluation 

1.  What  are  the  categories/factors  of  assessment  used  for  evaluating  past  performance? 
How  were  these  categories  constructed  (i.e.  industry  standard,  from  within)?  We  look  at 
quality,  acceptance  rate,  on-time  delivery  and  we  look  at  customer  satisfaction. 

2.  Is  your  rating  system  quantitative  (e.g.  late  delivery  in  terms  of  #  of  days)  or 
qualitative  (e.g.  quality  performance)?  Our  system  is  quantitative  in  terms  of  quality, 
acceptance  rate,  on-time  delivery.  It  is  subjective  or  qualitative,  if  you  will,  in  terms  of 
customer  satisfaction. 

3.  How  many  days  on  average  does  it  take  you  to  award  a  contract  (e.g.  clock  starts 
when  a  requirement  or  funding  document  is  received)?  It’s  hard  to  say. 

4.  Has  your  pre-award  evaluation  process  changed  over  the  last  5  years?  Not 
significantly,  just  in  that  much  of  our  rating  system  has  been  automated. 

5.  Have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your  evaluation 
process?  Nothing  that  we  can  pinpoint. 

1C.  What  is  your  impression  of  the  current  DoD  past  performance  and  risk  assessment? 

I  think  DoD  does  a  good  job.  It  is  extremely  thorough  and  in  most  cases  fair.  Obviously, 
it  is  very  expensive,  but  understandably  so  in  terms  of  the  magnitude  of  some  of  the 
programs. 

6.  Does  your  company  receive  feedback  from  other  business  partners  concerning  your 
own  past  performance  evaluation  (e.g.  how  you  are  perceived  or  rate  with  other  firms)? 
Yes,  they  submit  feedback  within  our  system. 

7.  Would  you  categorizes  most  of  your  contracts  fixed-price,  incentive  based,  or  cost- 
plus?  I  would  categorize  most  of  our  contract  as  fixed-price. 

8.  Is  the  importance  of  price  or  financial  performance  tailored  for  each  acquisition? 

Yes,  the  importance  of  price  depends  on  the  complexity  of  the  buy.  If  we  are  buying 
OEM  parts  then  price  is  most  important.  If  we  are  developing  a  new  system  then  we  are 
looking  at  total  cost.  This  includes  ownership,  maintenance,  and  quality  concerns. 

9.  Is  price  the  most  important  factor  in  evaluation?  No 
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Post- Award  Evaluation 


1 .  How  is  post-award  evaluation  maintained  by  your  firm? 

We  rate  suppliers  on  their  perfonnance  on  specific  procurement  and  we  give  them  a 
rating  across  the  board.  It  lets  them  know  how  they’re  doing. 

2.  Is  the  process  automated  or  manual?  What  is  the  role  of  purchasing  personnel  in  this 
process?  The  process  is  semi-automated.  The  database  requires  manual  input,  but  the 
output  can  be  pulled  automatically.  Purchasing’s  role  is  very  limited.  They  simply 
retrieve  information  when  they  need  it  to  assess  a  supplier’s  capability. 

2C.  What  is  your  impression  of  third-party  rating  systems  (e.g.  OpenRatings)?  I 
wouldn’t  feel  comfortable  putting  all  my  decisions  in  their  hands.  I  have  a  lot  of 
confidence  in  the  system  we  have  developed  in-house. 

3C.  If  your  process  is  automated,  who  designed  the  software  or  what  third-party 
company  maintains  this  database?  We  developed  the  software  in-house. 

3 .  How  much  time  (hours,  days)  is  required  to  complete  a  post-award  evaluation?  How 
often  is  it  perfonned  (never,  monthly,  quarterly)?  Not  much  time  is  required.  I  would 
say  30  minutes  to  compile  the  data.  The  information  is  then  compiled  into  a  12  month 
moving  average  for  each  site.  It  is  a  monthly  rating. 

4.  Does  your  evaluation  after  award  affect  award-fees  or  bonuses  given  for  above 
average  performance?  I’m  not  sure  if  it  applies. 

5.  What  factors  do  you  perceive  will  affect  the  success  or  failure  of  a  contract?  Process 
Capability  is  critical.  We  are  also  extremely  concerned  with  how  a  supplier  manages 
their  sub-tiers.  We  often  find  that  this  is  the  most  common  reason  for  a  supplier 
experiencing  delays. 

General  Questions 

1.  How  would  you  describe/define  strategic  sourcing?  Does  it  apply  before  and  after 
award?  We  have  a  strategic  sourcing  function  at  Boeing.  I  would  say  that  is  how  an 
organization  manages  their  suppliers.  It  is  the  ability  to  find  out  who  your  suppliers  are 
and  to  determine  who  is  a  good  supplier.  Finally,  I  would  say  that  strategic  sourcing  is 
helpful  in  detennining  whether  you  make  an  item  or  do  you  buy  it. 

4C.  Do  you  maintain  one  rating  for  each  supplier  which  covers  all  contracts  or  are  they 
given  a  rating  which  is  based  on  their  performance  on  one  particular  contract?  Our 
ratings  cover  all  contracts.  It  is  a  roll-up.  We  are  investigating  ways  to  break  it  down  to 
each  individual  contract  at  this  time. 
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2.  What  3  things  could  you  have  done  better  in  pre-award  and  post- award  evaluations 
knowing  what  you  know  now?  We  could  stand  to  improve  our  preferred  supplier 
program  in  house.  The  system  needs  to  be  institutionalized. 

3.  How  much  training  is  given  in-house  regarding  supplier  evaluations?  We  give  3  and 
half  to  4  hours  of  pre-award  training  on  source  selections.  We  also  give  4  hours  of 
training  to  cover  the  preferred  supplier  program.  This  addresses  the  post-award  phase. 

4.  Does  your  firm  use  multi-functional  teams  when  performing  pre-award  and  post¬ 
award  evaluations  or  do  different  departments  (e.g.  engineering,  purchasing,  marketing) 
act  separately?  Yes,  we  do  have  an  IPT.  It  is  a  matrixed  structure.  Especially  if  they’re 
involved  in  a  source  selection. 

5.  How  many  individuals  typically  participate  in  a  pre-award  or  post-award  evaluation 
(i.e.  $1M,  $50M,  &  $100M+  acquisitions)?  This  will  vary.  7  people  if  I  had  to  say  off 
the  top  of  my  head.  They’re  from  different  functions. 

6.  Would  you  characterize  you  relationships  with  suppliers  short-tenn  or  long-term?  I 
would  like  to  think  that  our  relationship  has  improved.  The  fact  that  we  have  preferred 
supplier  process  and  a  rating  system  in  place  it  has  helped  improved  supplier 
relationships.  Our  development  program  of  suppliers  has  received  feedback  that  we  work 
well  with  our  suppliers  on  the  military  side.  I  can’t  speak  of  the  commercial  side. 

7.  Has  your  relationship  with  your  suppliers  improved  or  suffered  because  of  your 
evaluation  procedures?  I  think  it  has  improved.  The  suppliers  feel  that  we  are  fair.  On 
the  commercial  side  I  think  it’s  just  the  opposite.  McDonnell  Douglas  had  a  history  of 
working  with  suppliers  as  partners.  There  is  a  heritage  there. 

5C.  Does  your  company  do  business  in  the  aircraft  or  manufacturing  industries?  Yes 

6C.  What  regulations  does  your  company  have  which  outline  the  pre-award  evaluation 
process?  We  have  PROs  (Enterprise  procedures)  that  people  are  trained  on.  We  have 
PRO  for  supplier  perfonnance  measurement.  The  PRO  number  is  3275.  We  have 
another  PRO  for  supplier  certification  1113  is  9  pages.  We  have  a  PRO  for  source 
selection.  We  have  a  PRO  for  strategic  sourcing  that  is  approximately  7  pages.  All  of 
these  PROs  help  us  to  document  what  our  process  is.  It  helps  us  to  train  our  people 
accordingly.  Typically,  a  PRO  should  only  be  2  pages.  But  if  you’re  trying  to  get  it 
across  all  of  Boeing,  you  need  to  cover  various  scenarios.  Most  are  under  30  pages. 
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(United  Air  Lines) 


Interview  Questions  for  Contractor/Program  Office  Personnel 


Contractor 

Pre- Award  Evaluation 

1.  What  are  the  categories/factors  of  assessment  used  for  evaluating  past  performance? 
How  were  these  categories  constructed  (i.e.  industry  standard,  from  within)?  If  it’s  a  new 
supplier  for  example  computer  products:  I  know  we’ll  need  6,000  for  airport  stations. 

The  people  in  information  systems  have  a  role,  the  people  in  purchasing  have  a  role,  the 
users  have  a  role,  and  the  managers  will  decide.  When  I  choose  a  new  company  I  want  a 
know  they’re  financially  viable,  if  they  have  on-time  delivery,  their  proximity,  and 
technical  performance. 

2.  Is  your  rating  system  quantitative  (e.g.  late  delivery  in  terms  of  #  of  days)  or 
qualitative  (e.g.  quality  performance)?  There  is  both.  Price  is  quantitative.  The  location 
of  the  supplier  and  responsiveness  is  qualitative. 

3.  How  many  days  on  average  does  it  take  you  to  award  a  contract  (e.g.  clock  starts 
when  a  requirement  or  funding  document  is  received)?  United  Airlines  is  one  of  the 
largest  telcomm  users  in  the  world.  A  telcomm  bid  could  take  8  months.  A  bid  for 
plastic  products  (polybags — for  skis  and  golf  clubs)  was  done  reverse  auction  and 
completed  in  1  afternoon,  although  the  specifications  were  clear  up-front.  It  depends  on 
the  complexity  of  the  buy. 

4.  Has  your  pre-award  evaluation  process  changed  over  the  last  5  years?  I  would.  It 
used  to  be  the  best  price.  It  is  no  longer  like  that. 

5 .  Have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your  evaluation 
process?  We  have  saved  money  by  reverse  auctions.  People  are  able  to  go  down  lower 
in  price  with  reverse  auctions  and  competitions  become  intense.  We  are  now  looking  at 
overall  costs  of  maintaining  the  item. 

1C.  What  is  your  impression  of  the  current  DoD  past  performance  and  risk  assessment? 
My  first  impression  is  the  $400  toilet  seat  or  the  $200  hammer.  It  is  very  difficult  to  deal 
with  DoD  at  times.  There  is  some  companies  that  do  not  want  to  deal  with  DoD  because 
of  the  administrative  burden. 

6.  Does  your  company  receive  feedback  from  other  business  partners  concerning  your 
own  past  performance  evaluation  (e.g.  how  you  are  perceived  or  rate  with  other  firms)? 

In  all  honesty  a  lot  of  times  the  people  that  don’t  get  the  business,  we  are  reluctant  to  give 
them  that  information.  The  reason  is  that  I  want  to  get  that  buy  completed.  The  last  thing 
the  buyers  want  to  do  is  get  in  to  great  detail  as  to  why  they  didn’t  win  the  bid. 


89 


7.  Would  you  categorizes  most  of  your  contracts  fixed-price,  incentive  based,  or  cost- 
plus?  Fixed-price  and  cost-plus  arrangements,  I  have  a  cost-plus  contract  with  Hewlett- 
Packard  for  personal  computers. 

8.  Is  the  importance  of  price  or  financial  performance  tailored  for  each  acquisition? 

Yes.  On  polybags  or  cans  of  coke  price  is  especially  important.  But,  when  I’m  dealing 
with  a  service  price  is  not  the  most  important. 

9.  Is  price  the  most  important  factor  in  evaluation?  Depends  on  the  complexity.  The 
initial  price  might  be  fine,  but  if  I’m  having  cans  of  coke  delivered  from  California  to 
Illinois  I’ll  get  killed  on  shipping. 

Post- Award  Evaluation 

1 .  How  is  post-award  evaluation  maintained  by  your  firm?  Unfortunately  to  some  extent 
a  lot  of  times  I  don’t  even  know  when  a  delivery  hasn’t  been  made  because  of  a  buffer 
stock  in  the  Denver  warehouse.  There’s  other  items  such  as  headsets  from  China,  we 
have  to  worry  about  the  efficiency  in  getting  us  the  delivery,  I  don’t  think  we  do 
statistical  analysis,  but  I  do  believe  the  individual  buyers  manage  their  purchases.  I  do 
know  that  we  look  to  see  that  invoices  are  correct.  I  attended  a  briefing  by  Hewlett 
Packard  in  Grenoble,  France.  The  person  in  charge  was  using  a  supplier  ratings  system. 

2.  Is  the  process  automated  or  manual?  What  is  the  role  of  purchasing  personnel  in  this 
process?  It  is  a  manual  process.  I  know  that  Dell  is  doing  the  automated  supplier  ratings. 
They  want  their  suppliers  right  next  door  so  they  can  track  them.  The  purchasing  role  is 
informal.  It  gets  back  to  the  end  user  buying  the  product.  If  I’m  not  hearing  complaints 
from  them,  I  know  that  they’re  satisfied. 

2C.  What  is  your  impression  of  third-party  rating  systems  (e.g.  OpenRatings)?  I  have 
heard  of  OpenRatings.  We  do  not  use  a  system  like  that.  I  think  in  different  financial 
conditions  I  would  absolutely  look  at  this.  It  would  make  my  job  easier. 

3C.  If  your  process  is  automated,  who  designed  the  software  or  what  third-party 
company  maintains  this  database?  We  do  have  a  system  from  Peregrine  for  asset 
tracking.  The  system  can  do  this.  I  just  don’t  know  if  we  are  using  this  system  to  its  full 
capability. 

3.  How  much  time  (hours,  days)  is  required  to  complete  a  post-award  evaluation?  How 
often  is  it  performed  (never,  monthly,  quarterly)?  It  is  not  done. 

4.  Does  your  evaluation  after  award  affect  award-fees  or  bonuses  given  for  above 
average  performance?  No  rating  system. 

5.  What  factors  do  you  perceive  will  affect  the  success  or  failure  of  a  contract? 
Management  of  the  supplier  is  extremely  important.  Often  times  the  supplier  has  free 
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roam  to  your  user,  the  user  becomes  pre-disposed  and  your  hands  are  tied  by  the 
specification  created.  The  requirement  should  be  clearly  defined  up-front. 

General  Questions 

1.  How  would  you  describe/define  strategic  sourcing?  Does  it  apply  before  and  after 
award?  That’s  what  our  department  is  called.  It  gets  into  the  issue  of  how  purchasing 
has  changed,  it  essentially  looking  at  more  than  price.  It  means  establishing  long-term 
relationships  that  create  long-term  value  for  the  company.  It  applies  before  and  after 
award. 

4C.  Do  you  maintain  one  rating  for  each  supplier  which  covers  all  contracts  or  are  they 
given  a  rating  which  is  based  on  their  performance  on  one  particular  contract?  No 
ratings.  Our  company  has  that  Perigrine  system  that  has  the  capability  to  run  the  reports 
to  see  that  this  product  was  received  on  this  date.  However,  we  haven’t  used  it  long 
enough  to  know  how  successful  its  been.  If  I  had  the  system  I  would  want  to  know  on  a 
particular  contract.  When  I  deal  with  a  company  like  IBM  or  Hewlett-Packard  for 
something  like  EZ  Pass  (Electronic  Boarding  Pass)  I  would  want  to  know  what  IBM 
division  actually  did  the  EZ  Pass  and  how  well  they  did.  Our  suppliers  have  many 
different  divisions  and  locations. 

2.  What  3  things  could  you  have  done  better  in  pre-award  and  post- award  evaluations 
knowing  what  you  know  now?  I  would  try  to  make  sure  the  specifications  are  more 
clearly  defined. 

3.  How  much  training  is  given  in-house  regarding  supplier  evaluations?  We  give  each 
new  employee  8  hours  of  training  and  there  other  classes  to  teach  purchasing  agents 
about  negotiation,  software,  and  legal  issues  that  amount  to  40  hours  per  year. 

4.  Does  your  firm  use  multi-functional  teams  when  performing  pre-award  and  post¬ 
award  evaluations  or  do  different  departments  (e.g.  engineering,  purchasing,  marketing) 
act  separately?  Yes  we  use  multi-disciplined  teams  that  all  work  in  the  same  building. 

5.  How  many  individuals  typically  participate  in  a  pre-award  or  post- award  evaluation 
(i.e.  $1M,  $50M,  &  $100M+  acquisitions)?  If  we  had  a  $50M  buy,  I  would  assume  that 
there  would  be  about  8  core  individuals  involved  in  the  buy. 

6.  Would  you  characterize  you  relationships  with  suppliers  short-term  or  long-tenn? 

We  certainly  hope  and  expect  that  our  relationships  are  more  and  more  long-term. 

7.  Has  your  relationship  with  your  suppliers  improved  or  suffered  because  of  your 
evaluation  procedures?  Our  relationships  have  improved. 

5C.  Does  your  company  do  business  in  the  aircraft  or  manufacturing  industries?  Yes 
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6C.  What  regulations  does  your  company  have  which  outline  the  pre-award  evaluation 
process?  We  have  a  very  thorough  set  of  regulations.  They  explain  the  agreements  that 
are  in  place.  We  use  web  regulations  that  are  probably  100  pages  in  length.  However, 
only  15  pages  of  that  are  purchasing  specific. 
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(UTC) 


Interview  Questions  for  Contractor/Program  Office  Personnel 


Contractor 

Pre- Award  Evaluation 

1.  What  are  the  categories/factors  of  assessment  used  for  evaluating  past  performance? 
How  were  these  categories  constructed  (i.e.  industry  standard,  from  within)?  We  have  8 
factors  we  look  at.  We  look  at  their  delivery,  quality,  financial  performance,  lean 
manufacturing  initiatives,  engineering  initiatives,  manufacturing  capabilities, 
manufacturing  capacity,  and  open  ratings. 

2.  Is  your  rating  system  quantitative  (e.g.  late  delivery  in  terms  of  #  of  days)  or 
qualitative  (e.g.  quality  performance)?  The  ratings  system  is  both.  The  Quantatitve  is 
delivery,  quality,  and  financial/cost  competitiveness.  The  Qualitative  is  lean 
manufacturing  initiatives,  engineering  initiatives,  manufacturing  capabilities,  and 
manufacturing  capacity. 

3.  How  many  days  on  average  does  it  take  you  to  award  a  contract  (e.g.  clock  starts 
when  a  requirement  or  funding  document  is  received)?  It  depends  on  how  motivated  we 
are.  I  would  say  for  a  $50M  contract  would  be  6  to  9  months. 

4.  Has  your  pre-award  evaluation  process  changed  over  the  last  5  years?  Not 
significantly.  We’ve  automated  things  but  we’re  evaluating  the  same  criteria. 

5 .  Have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your  evaluation 
process?  $  IB  Savings  on  the  product  side  to  Wall  Street  through  200 1 .  On  the  non¬ 
product  side  we  are  on  track  for  a  $700M  savings  by  the  year  2004. 

1C.  What  is  your  impression  of  the  current  DoD  past  performance  and  risk  assessment? 
No  impression  of  the  government. 

6.  Does  your  company  receive  feedback  from  other  business  partners  concerning  your 
own  past  performance  evaluation  (e.g.  how  you  are  perceived  or  rate  with  other  firms)? 
Y es,  they  call  us  they  e-mail.  They  work  through  their  commodity  manager  or  their 
management.  If  suppliers  get  good  ratings  we  don’t  hear  from  them,  if  they  get  bad 
ratings  they’ll  contact  us. 

7.  Would  you  categorize  most  of  your  contracts  fixed-price,  incentive  based,  or  cost- 
plus?  Fixed-Price  with  regressive  clause  that  decreases  the  cost  each  year  after. 

8.  Is  the  importance  of  price  or  financial  performance  tailored  for  each  acquisition? 
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Yes,  financial  performance  and  cost  of  the  product  is  a  key  driver.  If  we’re  looking  at 
buying  10,000  screws  the  price  is  most  important.  If  we’re  buying  an  engine  we’re  less 
concerned  about  price  but  more  concerned  about  the  future  investment. 

9.  Is  price  the  most  important  factor  in  evaluation?  My  personal  opinion  is  no.  The 
corporate  by  line  is  probably  yes. 

Post- Award  Evaluation 

1 .  How  is  post- award  evaluation  maintained  by  your  firm?  We  have  a  monthly 
scorecard  that  is  published  on  the  website.  All  of  the  six  divisions  at  UTC  roll  up  their 
receipts  for  their  suppliers  and  it  is  consolidated  at  corporate.  The  scorecard  contains 
quality  and  delivery  infonnation  only. 

2.  Is  the  process  automated  or  manual?  What  is  the  role  of  purchasing  personnel  in  this 
process?  It’s  a  combination  of  both.  It’s  automated  out  of  the  business  units  but  it’s  a 
manual  effort  to  consolidate  it.  We  have  a  lot  of  legacy  systems  still.  Purchasing  is  the 
first  line  of  defense  for  phone  calls,  they  don’t  play  a  role  in  getting  the  data  published. 

2C.  What  is  your  impression  of  third-party  rating  systems  (e.g.  OpenRatings)? 

Very  good.  We  leverage  it  internally.  When  we  look  at  performance  there  are  different 
pieces  we  look  at.  There  are  some  things  we  don’t  have  access  to.  The  Dun  &  Bradstreet 
information  from  OpenRatings  is  very  helpful.  We  don’t  put  100%  faith  in  OpenRatings 
but  we  do  use  it  quite  significantly  to  drive  internal  activities. 

3C.  If  your  process  is  automated,  who  designed  the  software  or  what  third-party 
company  maintains  this  database?  Home  grown  system  managed  out  of  our  Supplier 
Quality  Organization. 

3.  How  much  time  (hours,  days)  is  required  to  complete  a  post-award  evaluation?  How 
often  is  it  performed  (never,  monthly,  quarterly)?  Pulling  the  data  at  the  division  level  is 
automated.  The  information  is  automatically  taken  from  the  purchase  order,  it  includes 
delivery,  quantity,  and  inspection  information.  When  the  information  reaches  the 
corporate  level  the  roll-up  it  takes  minimal  time  to  roll-up. 

4.  Does  your  evaluation  after  award  affect  award-fees  or  bonuses  given  for  above 
average  performance?  We  don’t  give  bonuses  although  we  give  plaques  to  recognize 
good  suppliers. 

5.  What  factors  do  you  perceive  will  affect  the  success  or  failure  of  a  contract?  We  find 
that  we  have  the  most  success  with  suppliers  that  practice  Lean  Manufacturing.  Failure  is 
often  found  with  companies  that  give  the  best  price  but  don’t  necessarily  have  the 
capability.  That  is  probably  because  we’ve  given  them  an  impossible  challenge.  It  could 
be  bad  drawings  and  bad  specifications. 
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General  Questions 

1.  How  would  you  describe/define  strategic  sourcing?  Does  it  apply  before  and  after 
award?  Strategic  sourcing  consists  of  commodity  management  teams  responsible  for  the 
strategic  direction  of  the  supply  base.  The  goal  is  to  establish  as  much  commonality 
between  the  divisions.  The  products  are  often  similar  and  can  be  consolidated  to  create  a 
common  approach.  We  want  to  work  with  strategic  partners. 

4C.  Do  you  maintain  one  rating  for  each  supplier  which  covers  all  contracts  or  are  they 
given  a  rating  which  is  based  on  their  performance  on  one  particular  contract? 

We  have  a  scorecard  that  allows  you  to  see  a  roll-up  for  all  contracts.  We  can  get  to  the 
data  individually  but  we  don’t  look  at  it  that  way. 

2.  What  3  things  could  you  have  done  better  in  pre-award  and  post- award  evaluations 
knowing  what  you  know  now?  We  could  improve  in  how  we  collect  data.  Our  legacy 
systems  need  to  be  upgraded.  We  are  trying  to  detennine  who  it  is  that  we  spend  what 
with.  We  could  also  do  better  in  supplier  evaluations. 

3.  How  much  training  is  given  in-house  regarding  supplier  evaluations?  It  is  a  2  day  16 
hours  of  training  that  discusses  the  8-step  sourcing  process.  We  offer  them  additional 
training  called  Achieving  Competitive  Excellence  (ACE).  It  consists  of  lean 
manufacturing  principles.  On  average  people  receive  16-24  hours  of  training. 

4.  Does  your  firm  use  multi-functional  teams  when  performing  pre-award  and  post¬ 
award  evaluations  or  do  different  departments  (e.g.  engineering,  purchasing,  marketing) 
act  separately?  Yes,  our  commodity  teams  are  cross-functional.  Procurement,  Quality, 
Finance,  Manufacturing  Engineering,  and  Design  Engineering. 

5.  How  many  individuals  typically  participate  in  a  pre-award  or  post- award  evaluation 
(i.e.  $1M,  $50M,  &  $100M+  acquisitions)?  I  would  say  there  are  4  to  8  core  people  from 
each  functional  and  4  to  10  people  used  for  support. 

6.  Would  you  characterize  you  relationships  with  suppliers  short-term  or  long-tenn? 
Long-Term 

7.  Has  your  relationship  with  your  suppliers  improved  or  suffered  because  of  your 
evaluation  procedures?  Our  relationship  really  hasn’t  changed.  The  OpenRatings 
systems  that  is  coming  out  online  the  company  will  self  evaluate.  We  expect  this  to  be 
positive. 

5C.  Does  your  company  do  business  in  the  aircraft  or  manufacturing  industries?  Yes 

6C.  What  regulations  does  your  company  have  which  outline  the  pre-award  evaluation 
process?  The  8-step  sourcing  process  guide  is  approximately  10  to  20  pages  long. 
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(C-130  Avionics  Modernization  Program) 

USAF  Program  Office  Personnel 

Pre- Award  Evaluation 

1 .  What  are  the  categories  of  assessment  used  for  evaluating  past  performance?  How 
were  these  categories  constructed  (i.e.  USAF/AFMC  Guide,  DoD  Past  Perfonnance 
Guide,  from  within)?  We  essentially  used  the  CPARS  Structure.  It  includes  Technical 
Performance,  Program  Management,  Subcontractor  Management,  Systems  Engineering, 
and  Cost  Performance  on  relevant  contracts. 

2.  Is  your  rating  system  quantitative  (e.g.  late  delivery  in  tenns  of  #  of  days)  or 
qualitative  (e.g.  quality  performance,  technical  performance)?  The  rating  system  is 
extremely  qualitative.  We  made  every  effort  to  remain  subjective.  Objective  ratings  are 
hard  to  define  in  the  request  for  proposals  and  this  was  common  to  most  source  selections 
and  in  accordance  with  SAF/AQC  guidance. 

3.  How  many  days  on  average  does  it  take  you  to  award  a  contract  (e.g.  clock  starts 
when  a  requirement  or  funding  document  is  received)?  For  our  effort  the  program  was 
approximately  $2B  to  $3B.  The  process  took  more  than  a  year.  I  would  say  for  a  $50M 
effort  you  could  expect  it  to  take  6  months  to  a  year  from  start  to  finish. 

4.  Has  your  pre-award  evaluation  process  changed  over  the  last  5  years?  Not 
significantly.  The  source  selection  process  includes  the  same  infonnation. 

5.  Have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your  evaluation 
process?  No,  in  fact  the  acquisition  process  is  more  focused  toward  up-front  work. 

1G.  What  is  your  perception  of  the  FAR’s  restrictions  on  the  pre-award  evaluation 
process?  Not  all  that  restrictive  on  past  perfonnance.  Most  of  the  guidance  that  we  use 
for  source  selections  comes  from  command  and  AQC. 

6.  Does  your  office  receive  feedback  from  suppliers  regarding  past  perfonnance 
evaluation  (e.g.  source  selections)?  Yes,  we  received  feedback  during  de-briefings  prior 
to  award.  The  contractors  competing  for  the  effort  knew  where  they  stood  regarding  past 
performance  prior  to  the  final  de-brief. 

2G.  How  many  protests  have  you  received?  No  protests  were  received  on  this  effort, 
although  protests  do  occur  rather  frequently  when  so  much  is  at  stake.  I  think  in  this 
effort,  the  contractors  were  hesitant  to  protest  as  the  Joint  Strike  Fighter  was  upcoming. 

7.  Would  you  categorize  most  of  your  contracts  fixed-price,  incentive  based,  or  cost- 
plus?  Mixed,  but  a  majority  of  our  efforts  in  the  systems  environment  are  cost-plus  type 
arrangements.  Years  ago  there  was  a  push  to  go  fixed-price,  but  that  seems  to  have 
changed. 
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8.  Is  the  importance  of  price  or  financial  performance  tailored  for  each  acquisition? 

Yes,  it  is  usually  a  mix  between  cost,  technical,  and  past  performance  factors.  It  is 
difficult  to  assess  how  important  cost  is  on  each  evaluation.  I  can  say  that  it  is  more 
likely  to  be  a  discriminator  than  past  performance.  In  our  case  though,  I  think  past 
performance  made  a  difference. 

9.  Is  price  the  most  important  factor  in  evaluation?  No,  the  weights  of  cost,  technical, 
and  past  performance  vary  from  source  selection  to  source  selection. 

Post- Award  Evaluation 

1 .  How  is  post-award  evaluation  maintained  by  the  program  office?  Do  you  use  USAF 
or  AFMC  guides?  It  is  a  formal  process  driven  by  leadership.  There  are  typically  IPT’s 
that  cover  each  area  of  the  award  fee.  It  is  a  very  subjective  and  time  consuming  process. 
We  try  to  remain  subjective  as  opposed  to  objective  because  often  times  the  objective 
areas  will  be  the  only  areas  to  improve.  I  remember  Gen.  Kadish  under  the  C-17  program 
as  saying,  we  want  to  keep  it  subjective  for  this  reason.  I  think  it  worked  well. 

2.  What  is  the  role  of  purchasing  personnel  vs.  program  managers  in  this  process?  The 
contracting  officer  oversaw  adherence  to  the  contract  stipulations.  However,  it  was 
clearly  the  program  manager’s  responsibility  to  carry  out  the  award  fee  evaluation. 

3G.  If  this  process  could  be  automated  would  it  alter  your  perception  of  the  DoD 
evaluation  process?  Theoretically  you  could  automate  the  scoring.  But  if  you  did  you 
would  kill  the  fundamental  benefit  of  communication  between  the  teams. 

3 .  How  much  time  (hours,  days,  months)  is  required  to  complete  a  post-award 
evaluation?  How  often  is  it  perfonned  (never,  monthly,  quarterly)?  I  would  estimate  that 
the  award-fee  reviews  take  in  excess  of  100  man  hours.  In  my  programs,  I  have  seen 
Video  Teleconferences  that  are  conducted  every  3  months.  I  would  also  expect  6  month 
reviews  that  include  briefings  to  the  contractor.  This  VTCs  require  20  people  and  the 
there  are  various  meetings  and  TDYs  which  consume  8  program  and  contractor  personnel 
for  about  1  to  2  days  per  month. 

4.  Does  your  evaluation  allow  for  incentives  if  the  contractor  exceeds  the  requirement? 

Y es,  if  we  have  a  cost-plus  incentive  fee  arrangement. 

5 .  What  factors  do  you  perceive  will  affect  the  success  or  failure  of  a  contract?  I  would 
say  that  the  biggest  factor  of  any  program’s  success  or  failure  is  how  well  the  government 
and  contractor  team  communicates.  That  relationship  is  often  the  most  important. 

4G.  Does  the  program  office  currently  maintain  a  performance  file  for  each  of  its 
contractors?  Only  for  programs  that  have  award-fees.  We  also  have  CPARs  ratings  for 
some  system  contracts.  However,  we  are  only  capturing  files  contract  by  contract.  I 
would  like  to  see  an  overall  roll-up  so  that  we  could  assess  Boeing’s  St.  Louis  division. 
Currently,  there  is  no  means  for  doing  this. 
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General  Questions 

1.  How  would  you  describe/define  strategic  sourcing?  Does  it  apply  before  and  after 
award?  Not  heard  of  the  term,  but  it  would  say  it’s  sourcing  at  the  higher  level.  I  would 
think  that  it  would  mean  building  systems  around  common  platforms.  I’m  currently 
involved  with  the  UAV  systems.  Essentially,  we  have  the  same  guts  that  can  be  modified 
or  upgraded  rather  easily. 

2.  What  3  things  could  you  have  done  better  in  pre-award  and  post- award  evaluations 
knowing  what  you  know  now?  Get  involved  with  DCMA  at  the  earliest  possible  date 
prior  to  a  source  selection.  They  know  how  one  particular  division  is  performing. 

3.  How  much  training  is  given  in-house  regarding  supplier  evaluations? 

An  hour  of  training  is  given  prior  to  source  selections.  I  do  not  know  of  any  specific 
training  which  addresses  supplier  evaluations.  I  would  assume  its  taught  at  the  initial 
acquisition  courses.  What  frightens  me  is  that  many  of  our  senior  acquisition  officials  at 
the  product  center  have  never  been  involved  in  a  source  selection  and  as  a  result  they 
don’t  have  the  experience.  There  are  people  that  they  can  go  to  for  a  advice,  but  we  need 
more  experience  within  the  program  offices. 

4.  Does  your  program  office  use  multi-functional  teams  when  perfonning  pre-award  and 
post-award  evaluations  or  do  different  functionals  (e.g.  engineering,  purchasing, 
marketing)  act  separately?  Yes,  we  use  the  matrixed  approach.  Many  functional  experts 
are  now  sitting  together.  This  seems  to  work  well. 

5.  How  many  individuals  typically  participate  in  a  pre-award  or  post- award  evaluation 
(i.e.  $1M,  $50M,  &  $100M+  acquisitions)?  I  would  say  20  people  for  a  $50M  program. 
Our  C-130  AMP  program  had  60  to  70  people. 

6.  Would  you  characterize  you  relationships  with  suppliers  short-term  or  long-tenn? 

I  would  said  the  relationship  with  major  primes  is  long-term.  However,  we  don’t  have 
any  relationship  with  subcontractors. 

7.  Has  your  relationship  with  your  suppliers  improved  or  suffered  because  of  your 
evaluation  procedures?  We’ve  lost  the  ability  to  seek  out  new  suppliers.  We  are 
becoming  more  and  more  dependent  on  our  prime  contractors.  Overall,  though,  our 
relationship  with  our  current  suppliers  has  probably  improved  as  the  number  of 
debriefings  of  their  performance  has  increased. 
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USAF  Program  Office  Personnel  (AFRL  Enterprise  Buying  System) 

Pre- Award  Evaluation 

1 .  What  are  the  categories  of  assessment  used  for  evaluating  past  performance?  How 
were  these  categories  constructed  (i.e.  USAF/AFMC  Guide,  DoD  Past  Perfonnance 
Guide,  from  within)?  We  assess  management,  technical,  functionality,  operations  and 
sustainment,  and  cost  performance  in  general.  These  criteria  were  constructed  from 
related  programs. 

2.  Is  your  rating  system  quantitative  (e.g.  late  delivery  in  tenns  of  #  of  days)  or 
qualitative  (e.g.  quality  performance,  technical  performance)?  Mixture  of  both.  We  use 
questionnaires  on  source  selections  and  often  times  we  make  the  Likert  Scale 
quantitative.  The  comments  are  qualitative  and  are  considered  subjective. 

3.  How  many  days  on  average  does  it  take  you  to  award  a  contract  (e.g.  clock  starts 
when  a  requirement  or  funding  document  is  received)?  I  would  say  that  from  start  to 
finish  it  takes  6  months  to  a  year. 

4.  Has  your  pre-award  evaluation  process  changed  over  the  last  5  years?  I  would  say 
that  the  PRAG  process  has  more  structure  that  it  had  5  years  ago.  There  are  guides  that 
you  can  use. 

5.  Have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your  evaluation 
process?  No 

1G.  What  is  your  perception  of  the  FAR’s  restrictions  on  the  pre-award  evaluation 
process?  I  don’t  feel  restricted  by  the  FAR.  Most  of  the  procedures  aren’t  in  the  FAR. 
The  policies  usually  come  out  of  headquarters  for  a  source  selection. 

6.  Does  your  office  receive  feedback  from  suppliers  regarding  past  perfonnance 
evaluation  (e.g.  source  selections)?  Yes,  contractors  are  able  to  give  us  feedback  during 
the  debrief. 

2G.  How  many  protests  have  you  received?  We  did  not  receive  a  protest  on  the 
Enterprise  Business  System.  But  yes,  we  do  get  protests  on  other  source  selections. 

7.  Would  you  categorize  most  of  your  contracts  fixed-price,  incentive  based,  or  cost- 
plus?  I  would  say  that  AFRL  does  predominantly  research  contracts.  Most  of  our 
contracts  are  cost-type.  However,  this  effort  was  a  mix  of  fixed-price  and  cost-plus. 

8.  Is  the  importance  of  price  or  financial  performance  tailored  for  each  acquisition? 

I  think  cost  is  highly  important.  It  is  often  not  the  most  important,  but  if  the  requirements 
are  well  defined,  it  should  be. 


99 


9.  Is  price  the  most  important  factor  in  evaluation?  No,  I  would  like  to  say  that  we  use 
the  best  value  approach.  This  meaning  that  technical  perfonnance,  cost  performance,  and 
past  performance  are  considered. 

Post- Award  Evaluation 

1 .  How  is  post-award  evaluation  maintained  by  the  program  office?  Do  you  use  USAF 
or  AFMC  guides?  Under  the  science  and  technology  arena  this  process  is  very  informal. 
This  makes  it  hard  to  compare  suppliers  down  the  road. 

2.  What  is  the  role  of  purchasing  personnel  vs.  program  managers  in  this  process? 

Most  of  the  purchasing  personnel  will  play  an  administrative  role.  They  are  there  to 
ensure  that  reports  are  received  on  time. 

3G.  If  this  process  could  be  automated  would  it  alter  your  perception  of  the  DoD 
evaluation  process?  I  would  say  that  there  is  a  lot  of  infonnation  that  can  not  be 
quantified  in  an  automated  rating.  Subjective  ratings  can  be  a  good  thing. 

3.  How  much  time  (hours,  days,  months)  is  required  to  complete  a  post-award 
evaluation?  How  often  is  it  perfonned  (never,  monthly,  quarterly)?  I’m  not  sure  that  this 
is  really  done  for  science  and  technology  contracts. 

4.  Does  your  evaluation  allow  for  incentives  if  the  contractor  exceeds  the  requirement? 
Very  little  incentives  are  included  in  our  contracts.  However,  that  sort  of  thing  goes  in 
cycles. 

5 .  What  factors  do  you  perceive  will  affect  the  success  or  failure  of  a  contract? 

Overall,  I  would  say  the  contractor  management  is  most  important.  You  can  tell  if  the 
contractor  is  on  top  of  things. 

4G.  Does  the  program  office  currently  maintain  a  performance  file  for  each  of  its 
contractors?  No,  not  currently. 

General  Questions 

1 .  How  would  you  describe/define  strategic  sourcing?  Does  it  apply  before  and  after 
award?  I’m  not  exactly  sure  what  it  means.  I  know  headquarters  mentions  it  during 
meetings,  but  it  hasn’t  filtered  down. 

2.  What  3  things  could  you  have  done  better  in  pre-award  and  post-award  evaluations 
knowing  what  you  know  now?  Got  an  earlier  start.  We  should  have  put  the 
questionnaires  in  the  request  for  proposal.  Additionally,  the  criteria  we  are  assessing 
should  be  more  aligned  to  the  questionnaires. 

3 .  How  much  training  is  given  in-house  regarding  supplier  evaluations? 


100 


A  couple  of  hours  of  training  is  given  on  EZ  Source.  This  is  a  software  program  that  aid 
the  source  selection  team. 

4.  Does  your  program  office  use  multi-functional  teams  when  perfonning  pre-award  and 
post-award  evaluations  or  do  different  functionals  (e.g.  engineering,  purchasing, 
marketing)  act  separately?  Yes,  we  use  the  IPT  approach. 

5.  How  many  individuals  typically  participate  in  a  pre-award  or  post- award  evaluation 
(i.e.  $1M,  $50M,  &  $100M+  acquisitions)?  For  our  program,  which  was  $15M  it  took  15 
to  20  people  about  6  weeks  to  fully  evaluate  the  suppliers. 

6.  Would  you  characterize  you  relationships  with  suppliers  short-term  or  long-tenn? 

I  would  say  it’s  a  mix.  An  example  of  a  long-tenn  supplier  in  the  labs  is  in  the 
propulsion  division.  An  example  of  short-term  is  the  Small  Business  Innovative 
Research  (SBIR)  contractors. 

7.  Has  your  relationship  with  your  suppliers  improved  or  suffered  because  of  your 
evaluation  procedures?  I  would  like  to  think  that  it  will  improve  based  on  the  feedback 
we  give  during  debriefings.  However,  those  that  do  not  win  are  often  more  upset  and  do 
not  understand  the  process. 
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(FedEx) 


Interview  Questions  for  Contractor/Program  Office  Personnel 

Contractor 

Pre- Award  Evaluation 

1.  What  are  the  categories/factors  of  assessment  used  for  evaluating  past  performance? 
How  were  these  categories  constructed  (i.e.  industry  standard,  from  within)?  We  use  a 
scorecard  system  that  evaluates  price  trends,  services,  financial  performance,  on-time 
delivery  performance,  certification,  reporting,  invoice  discrepancy  rate,  receipt 
discrepancy  rate,  cycle-time  performance,  and  globalization  capability.  It  was  designed 
from  within. 

2.  Is  your  rating  system  quantitative  (e.g.  late  delivery  in  terms  of  #  of  days)  or 
qualitative  (e.g.  quality  performance)?  On-time  delivery  is  quantitative  as  well  as 
discrepancy  rates,  cycle -time  performance,  and  financial  performance.  The  rest  are 
quantitative. 

3.  How  many  days  on  average  does  it  take  you  to  award  a  contract  (e.g.  clock  starts 
when  a  requirement  or  funding  document  is  received)?  $50M  contract  would  take  5  or  6 
months. 

4.  Has  your  pre-award  evaluation  process  changed  over  the  last  5  years?  The  process 
has  changed  to  include  more  cross  functional  leads  and  towards  a  supply  chain 
management  focus.  We  are  also  engaging  more  diverse  (minority)  suppliers. 

5.  Have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your  evaluation 
process?  We  have  quantified  a  5  to  10%  savings  using  reverse  auctions  with  E-sourcing. 
We  are  involved  in  a  program  called  AeroExchange  that  manages  the  auctions  for  us. 

1C.  What  is  your  impression  of  the  current  DoD  past  performance  and  risk  assessment? 
No  impression  of  the  DoD  past  performance  process. 

6.  Does  your  company  receive  feedback  from  other  business  partners  concerning  your 
own  past  performance  evaluation  (e.g.  how  you  are  perceived  or  rate  with  other  firms)? 
We  do  not  receive  much  feedback  from  other  partners.  We  try  to  be  as  clear  as  possible 
with  our  scorecard  system. 

7.  Would  you  categorize  most  of  your  contracts  fixed-price,  incentive  based,  or  cost- 
plus?  Fixed-Price 

8.  Is  the  importance  of  price  or  financial  performance  tailored  for  each  acquisition?  Yes, 
the  weight  of  price  depends  on  the  strategic  importance  of  each  buy. 
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9.  Is  price  the  most  important  factor  in  evaluation?  In  most  cases  it  is,  but  40%  of 
weighting  is  assigned  to  support  services. 

Post- Award  Evaluation 

1 .  How  is  post-award  evaluation  maintained  by  your  firm?  There  are  two  levels  of 
management.  We  have  Supply  Chain  Specialists  who  own  the  process  from  cradle  to 
grave.  We  also  have  sourcing  managers  who  own  the  larger  contracts.  They  produce 
ratings  on  a  monthly  basis  via  a  scorecard  on  each  supplier.  Additionally,  there  is  a 
supply  management  specialist  (SMS)  who  maintain  a  portfolio  on  each  company. 

Overall,  planners  and  buyers  are  consolidated  to  manage  the  supply  chain  focus.  We 
seem  to  have  lost  some  intelligence  about  the  buying  side  by  consolidating  the  buying 
and  planning  functions. 

2.  Is  the  process  automated  or  manual?  What  is  the  role  of  purchasing  personnel  in  this 
process?  A  Little  of  both.  The  number  of  purchase  orders  that  are  delivered  on-time  is 
automated.  There  is  still  manual  manipulation  of  data  when  discrepancies  are  discovered. 
It  takes  purchasing  30  minutes  to  an  hour  to  get  data.  If  there  are  discrepancies  it  can 
take  much  longer. 

2C.  What  is  your  impression  of  third-party  rating  systems  (e.g.  OpenRatings)?  I  don’t 
feel  that  I  would  be  able  to  trust  the  data  as  much  as  the  data  of  our  own  systems.  AirBus 
does  this  but  doesn’t  also  have  great  data. 

3C.  If  your  process  is  automated,  who  designed  the  software  or  what  third-party 
company  maintains  this  database?  The  scorecard  system  was  built  in-house  and  is  posted 
on  the  website. 

3.  How  much  time  (hours,  days)  is  required  to  complete  a  post-award  evaluation?  How 
often  is  it  performed  (never,  monthly,  quarterly)?  It  takes  a  couple  of  minutes.  The  time 
required  is  generally  associated  with  data  retrieval. 

4.  Does  your  evaluation  after  award  affect  award-fees  or  bonuses  given  for  above 
average  performance?  There  are  some  incentives  if  contractors  perfonn  exceptionally 
well  on  the  scorecard.  It  depends  if  it  is  included  in  the  contract  terms. 

5.  What  factors  do  you  perceive  will  affect  the  success  or  failure  of  a  contract? 

Examples  include:  Quality  Management  Practices,  Documentation  and  self-audit, 
process/manufacturing  capability,  management  of  the  firm,  design  and  development 
capabilities,  and  cost  reduction  capabilities:  There  are  two  things.  First,  quality  is  a 
huge  emphasis  at  FedEx.  We  must  ensure  that  the  product  is  delivered  and  is  acceptable. 
If  not  we  have  to  deal  with  refunds  or  repairs.  The  second  item  is  service  related.  When 
companies  are  bought  out  or  merge  with  other  companies  it  can  cause  problems  down  the 
road. 
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General  Questions 

1.  How  would  you  describe/define  strategic  sourcing?  Does  it  apply  before  and  after 
award?  It  is  making  sure  that  you  identify  your  requirements  up-front  and  early.  It  also 
includes  optimizing  you  supply  base  to  decrease  the  number  of  suppliers  that  you  partner 
with.  Yes,  it  applies  before  and  after  award. 

4C.  Do  you  maintain  one  rating  for  each  supplier  which  covers  all  contracts  or  are  they 
given  a  rating  which  is  based  on  their  performance  on  one  particular  contract?  We  have 
one  rating  for  each  contract,  however,  we  roll-up  score  cards  if  the  company  has  several 
divisions. 

2.  What  3  things  could  you  have  done  better  in  pre-award  and  post- award  evaluations 
knowing  what  you  know  now?  Make  sure  the  source  of  supply  a  little  more  in-depth. 

3.  How  much  training  is  given  in-house  regarding  supplier  evaluations?  New  employees 
receive  40  hours  of  training  each  year.  After  that  employees  receive  20  hours  of  training 
per  year. 

4.  Does  your  firm  use  multi-functional  teams  when  performing  pre-award  and  post¬ 
award  evaluations  or  do  different  departments  (e.g.  engineering,  purchasing,  marketing) 
act  separately?  Yes,  we  use  integrated  product  teams  when  evaluating  suppliers. 

5.  How  many  individuals  typically  participate  in  a  pre-award  or  post- award  evaluation 
(i.e.  $1M,  $50M,  &  $100M+  acquisitions)?  I  would  say  there  is  a  core  team  of  about  6  to 
8  people  when  performing  an  evaluation  for  a  $50M  effort. 

6.  Would  you  characterize  you  relationships  with  suppliers  short-tenn  or  long-term?  Our 
relationships  are  becoming  more  and  more  long-term. 

7.  Has  your  relationship  with  your  suppliers  improved  or  suffered  because  of  your 
evaluation  procedures?  The  suppliers  like  our  scorecards.  It  provides  them  feedback  on 
a  more  consistent  basis.  Our  suppliers  are  now  asking  that  they  be  able  to  fill  out  a 
scorecard  on  us. 

5C.  Does  your  company  do  business  in  the  aircraft  or  manufacturing  industries?  Yes 

6C.  What  regulations  does  your  company  have  which  outline  the  pre-award  evaluation 
process?  There  is  a  fonnal  request  for  proposal  guide  which  is  30  pages  long.  Other 
information  can  be  found  on  the  website. 
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(Lockheed  Martin) 


Interview  Questions  for  Contractor/Program  Office  Personnel 

Contractor 

Pre- Award  Evaluation 

1.  What  are  the  categories/factors  of  assessment  used  for  evaluating  past  performance? 
How  were  these  categories  constructed  (i.e.  industry  standard,  from  within)?  We  assess 
quality,  delivery,  affordability,  and  management.  This  was  detennined  from  within. 

2.  Is  your  rating  system  quantitative  (e.g.  late  delivery  in  tenns  of  #  of  days)  or 
qualitative  (e.g.  quality  performance)?  Quality  and  delivery  are  quantitative. 
Affordability  and  management  are  qualitative.  Quality  and  Delivery  information  is  e- 
mailed  directly  to  the  supplier  on  a  monthly  basis.  The  qualitative  information  is  sent 
semi-annually.  It  is  sent  electronically  in  a  performance  feedback  report  to  225  major 
suppliers. 

3.  How  many  days  on  average  does  it  take  you  to  award  a  contract  (e.g.  clock  starts 
when  a  requirement  or  funding  document  is  received)?  I  can’t  answer  that. 

4.  Has  your  pre-award  evaluation  process  changed  over  the  last  5  years?  The  process  is 
generally  the  same  with  the  exception  of  the  rating  systems. 

5.  Have  you  been  able  to  quantify  any  savings  over  the  last  5  years  with  your  evaluation 
process?  Can’t  answer  that. 

1C.  What  is  your  impression  of  the  current  DoD  past  performance  and  risk  assessment? 
No  impression  of  the  DoD  past  performance  process. 

6.  Does  your  company  receive  feedback  from  other  business  partners  concerning  your 
own  past  performance  evaluation  (e.g.  how  you  are  perceived  or  rate  with  other  firms)? 
We  receive  feedback  from  them  at  our  annual  supplier  conference.  This  consists  of  our 
strategic  suppliers. 

7.  Would  you  categorize  most  of  your  contracts  fixed-price,  incentive  based,  or  cost- 
plus?  Not  for  certain. 

8.  Is  the  importance  of  price  or  financial  performance  tailored  for  each  acquisition?  Yes, 
because  we  are  now  moving  to  a  best  value  approach.  We  use  reverse  auctions  but  they 
are  still  best  value. 

9.  Is  price  the  most  important  factor  in  evaluation?  No. 

Post- Award  Evaluation 
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1 .  How  is  post-award  evaluation  maintained  by  your  firm?  We  use  the  ratings  that 
compile  on  each  supplier  and  we  use  them  for  future  award  determinations.  We  have  a 
STAR  supplier  program  to  recognize  excellent  performance  from  suppliers.  It  is  a  type 
of  award  that  most  suppliers  are  proud  of.  There  is  also  a  Supplier  Performance 
Evaluation  &  Action  Response  (SPEAR)  program  to  help  suppliers  recover  or  get  them 
up  to  speed. 

2.  Is  the  process  automated  or  manual?  What  is  the  role  of  purchasing  personnel  in  this 
process?  A  good  portion  of  it  is  automated.  The  monthly  portion  of  quality  and  delivery 
feedback  is  automated  via  e-mail.  The  semi-annual  data  is  somewhat  manual.  The 
buyers  complete  a  web  form  to  fill  out  the  quantitative  information. 

2C.  What  is  your  impression  of  third-party  rating  systems  (e.g.  OpenRatings)?  The 
system  at  Lockheed  Martin  is  already  so  automated  that  OpenRatings  would  not  provide 
much  benefit.  I  feel  that  OpenRatings  would  provide  that  most  benefit  to  a  small 
company  that  doesn’t  have  the  computer  programming  resources  of  a  large  firm  such  as 
Lockheed  Martin. 

3C.  If  your  process  is  automated,  who  designed  the  software  or  what  third-party 
company  maintains  this  database?  The  software  was  designed  internally  using  some 
COTS  software.  It  took  about  a  week  for  a  team  of  experts  within  the  company  to  design 
it. 

3 .  How  much  time  (hours,  days)  is  required  to  complete  a  post-award  evaluation?  How 
often  is  it  perfonned  (never,  monthly,  quarterly)?  The  monthly  reports  take  about  30 
minutes  to  complete.  The  semi-annual  reports  take  awhile  to  compile  but  to  batch  the 
information  requires  about  30  minutes. 

4.  Does  your  evaluation  after  award  affect  award-fees  or  bonuses  given  for  above 
average  performance?  It  depends  if  it  is  included  in  the  contract  terms  and  the  ratings 
certainly  won’t  be  the  only  consideration  given. 

5.  What  factors  do  you  perceive  will  affect  the  success  or  failure  of  a  contract? 

Examples  include:  Quality  Management  Practices,  Documentation  and  self-audit, 
process/manufacturing  capability,  management  of  the  firm,  design  and  development 
capabilities,  and  cost  reduction  capabilities:  Subcontracting  Management.  Lockheed 
Martin  is  pushing  its  suppliers  hard  to  manage  and  rate  their  suppliers. 

General  Questions 

1.  How  would  you  describe/define  strategic  sourcing?  Does  it  apply  before  and  after 
award?  It  is  not  a  fonnal  partnership  with  suppliers  but  a  level  of  trust  in  working 
together.  It  also  involves  stratifying  the  supply  base  by  top  level  managers. 

4C.  Do  you  maintain  one  rating  for  each  supplier  which  covers  all  contracts  or  are  they 
given  a  rating  which  is  based  on  their  performance  on  one  particular  contract?  It  is  an 
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overall  rating  for  a  particular  supplier.  But  it  can  be  broken  down  by  program  to  describe 
the  details  of  one  particular  effort. 

2.  What  3  things  could  you  have  done  better  in  pre-award  and  post-award  evaluations 
knowing  what  you  know  now?  Better  control  or  management  over  our  supplier’s 
suppliers  (subcontractor  management).  They  are  often  the  reason  for  a  delay. 

3.  How  much  training  is  given  in-house  regarding  supplier  evaluations?  There  is  3  hours 
of  training  given  to  new  employees  which  depicts  evaluation  procedures  at  Lockheed 
Martin. 

4.  Does  your  firm  use  multi-functional  teams  when  performing  pre-award  and  post¬ 
award  evaluations  or  do  different  departments  (e.g.  engineering,  purchasing,  marketing) 
act  separately?  Yes,  we  use  integrated  product  teams  when  evaluating  suppliers. 

5.  How  many  individuals  typically  participate  in  a  pre-award  or  post-award  evaluation 
(i.e.  $1M,  $50M,  &  $100M+  acquisitions)?  It  depends. 

6.  Would  you  characterize  you  relationships  with  suppliers  short-tenn  or  long-term?  I 
would  say  that  they  are  long-term. 

6.  Has  your  relationship  with  your  suppliers  improved  or  suffered  because  of  your 
evaluation  procedures?  The  suppliers  like  our  rating  systems.  They  will  ask 
questions  if  our  e-mails  are  late. 

5C.  Does  your  company  do  business  in  the  aircraft  or  manufacturing  industries?  Yes 

6C.  What  regulations  does  your  company  have  which  outline  the  pre-award  evaluation 
process?  There  are  buyer  checklists  on  the  web  to  ensure  that  each  buyer  follows  the 
same  process. 
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